Method and apparatus for linking designated portions of a received document image with an electronic address

ABSTRACT

A method and apparatus for receiving document images including portions linked to one or more electronic addresses. The linked portion of the document is identified using a predetermined visual attribute, such as bold-face text, or delimiters to mark the portion. The document image is then transmitted using, e.g., existing apparatus for transmitting images, such as a facsimile machine. An electronic address associated with the identified portion of the document is also transmitted using existing techniques, such as by touch-tone telephone. The address may be a voice telephone number, facsimile telephone number, World Wide Web address, or any other address with which communication can be established. At the receiving end, both the document image and electronic address are received. Pattern matching is performed on the document image to identify the portion with the predetermined attribute as a linked portion. The received electronic address is then correlated with the linked portion. When the document image is displayed on, e.g., a computer screen, the linked portion is visually identified so that the recipient can access the electronic address by, e.g., clicking on the portion using a mouse. Communication may then be initiated by the recipient with the entity associated with the electronic address.

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 08/475,204, filed Jun. 7, 1995, which is acontinuation-in-part of application Ser. No. 08/137,718, filed Oct. 15,1993, now U.S. Pat. No. 5,495,581, which is a continuation-in-part ofU.S. patent application Ser. No. 08/124,381, filed Sep. 17, 1993, whichis a continuation of U.S Ser. No. 07/918,150, filed Jul. 24, 1992, nowabandoned, which is a continuation of U.S. Ser. No. 07/840,808, filedFeb. 25, 1992, now abandoned.

1. Field of the Invention

The present invention relates to a method and apparatus forcommunicating documents with portions linked to electronic referenceinformation. More specifically, the present invention relates to amethod and apparatus for receiving document images and linking thereceived image with electronic reference information, wherein the imageis communicated using standard, existing equipment that otherwise wouldnot have this capability.

2. Background of the Invention

Known in the art are document management/recognition systems forfacilitating the archival storage and retrieval of documents. Some ofthese systems will reduce a document down to its constituent componentsin the course of processing the document for storage. Examples ofconstituent components are text fragments, graphics fragments, barcodes, etc. In order to enable the retrieval of any given document, orto permit an original document to be reconstructed from its parts, indexinformation may be saved so as to permit associations to be formedbetween attributes of the documents. However, such methods do not teachthe method and apparatus of the present invention.

Also known in the art are systems for manipulating symbolicrepresentations of text such as ASCII, EBCDIC, or the like, whereinembedded codes can be inserted at a position in the text to refer toanother position, or to other information associated with the positionof the embedded code. The disadvantage of these systems, however, isthat the document must be both created and accessed using softwarecapable of processing the embedded codes.

The document images of interest herein can reside on conventionalhardcopy media such as paper, and may be transmitted using nothing morethan ordinary facsimile equipment designed for transmitting ordinarymonochrome hardcopy documents. A distinction should be drawn betweenequipment sometimes referred to as "Desktop Fax" units and those called"PC-Fax" units. The former are designed to accept hardcopy documents asinput, which are scanned and then transmitted to a remote receivingstation. The latter operate on data already in electronic format whichreside in the memory of a computer; and, with respect to paper-basedhardcopy, require their conversion into electronic form beforeoperations of any kind can be performed on the contents of the hardcopy.Whereas the method taught by the present invention is not precluded fromuse with PC-Fax equipment, it is particularly useful in connection withordinary Desktop Fax equipment--a significant consideration in view ofthe limitations of such devices as compared to the capabilitiespossessed by the former. The present invention can thus be used by themost rudimentary of facsimile equipment, thereby imposing a minimal setof requirements on the document sender with respect to hardware andsoftware. In contrast, this is simply not possible with techniquescurrently known in the art. For example, techniques which rely on binaryfile transfer to communicate documents containing color transmit digitaldata files, as is evident from the name, and not document images;moreover, they mandate the use of a computer to process the data filebeing sent.

The present invention is also to be distinguished from systems directedat manipulating symbolic representations of document information, suchas ones where text occurs in ASCII form, rather than in the form of adocument image. Such systems cannot operate on information contained inhardcopy documents until the hardcopy documents have been opticallyscanned and converted into machine manipulatable code by OCR software.Only then can commands of any kind be embedded into the block ofsymbolic data. Furthermore, appropriate hardware such as a computer, andsoftware such as a text editor, must be available for manipulating thesymbolic data. Such systems do not enable conventional facsimileequipment to be used as they lack the capacity for symbolicmanipulation. Such systems clearly do not enable a sender having nothingmore than a conventional monochrome facsimile machine, a hardcopydocument, and a marking implement such as a pen to achieve the benefitsdescribed herein.

It is an object of the present invention to provide a method andapparatus for altering designated portions of a received document imagethat does not suffer from the drawbacks of prior art systems.

SUMMARY OF THE INVENTION

The present invention relates to a method and apparatus forcommunicating documents with portions linked to electronic referenceinformation. More specifically, the present invention relates to amethod and apparatus for receiving document images and linking thereceived image with electronic reference information, wherein the imageis communicated using standard, existing equipment that otherwise wouldnot have this capability. For example, a standard fax machine may beused to convey a document having a portion electronically linked toelectronic audio information.

The present invention comprises an apparatus for linking a document withreference information associated therewith comprising means forreceiving an electronic representation of a document image, saiddocument having a portion to be linked to electronic referenceinformation, wherein said portion is designated by a predeterminedattribute of the received document image; means for electronicallyscanning the electronic representation of the document image to locatesaid predetermined attribute of the document image; means foridentifying the linked portion of the document based on the location ofsaid predetermined attribute; means for acquiring the electronicreference information associated with the linked portion; means forcorrelating the linked portion of the document with the associatedelectronic reference information; and means for providing a pointer fromthe linked portion in the document to each piece of electronic referenceinformation associated therewith.

In one embodiment of the invention, the received document image may betransmitted using standard facsimile protocols and the electronicreference information comprises audio information transmitted usingstandard voice channel telephony.

In a further embodiment of the present invention, the electronicreference information comprises address information related to thedesignated portion of the document image. The recipient of the documentimage is provided with means for receiving the document image and theelectronic reference information comprising address information. Therecipient's means further comprises means for accessing the addressinformation and establishing a communications session with the addressor addresses associated with the designated portion of the document,which may, for example, include a voice telephone number, facsimilenumber, pager number, e-mail address, or a World Wide Web (WWW) address.

BRIEF DESCRIPTION OF THE DRAWINGS

A detailed description of preferred embodiments of the invention will bemade with reference to the accompanying drawings, wherein like numeralsdesignate corresponding parts in the several figures.

FIG. 1 illustrates a known copying apparatus in schematic form.

FIG. 2 depicts a preferred embodiment of the present invention mountedon top of the window of the copying apparatus illustrated in FIG. 1.

FIG. 3 illustrates a preferred embodiment of the image-formingcomponents of the present invention.

FIG. 4 illustrates a reflective layer that is disposed adjacent to ornear the image-forming element of FIG. 3.

FIG. 5 illustrates the structures shown in FIG. 4 with an image presenton the image-forming member (in this particular example, a set ofalphabetical characters assembled into the word "Hello").

FIG. 6 is a perspective view from below the copier window of thestructure in FIG. 5, when the illustrated embodiment of the presentinvention is mounted on the copy board of a copier.

FIG. 7 illustrates an embodiment of the invention that can be used withcopiers possessing moving copy-boards, which are common to many low cost"personal" copiers.

FIG. 8 illustrates an alternate embodiment to that depicted in FIG. 5 inwhich the image-forming element is of the emissive variety.

FIG. 9 illustrates an embodiment of the invention in which the signalrepresentative of the image to be printed is transmitted to theimage-forming member by an "intangible mechanism" such as an infra-redbeam, rf-signal, etc. (in contrast to a cable or electrical wire).

FIG. 10 illustrates one possible arrangement of photodetectors on animage forming tablet.

FIG. 11 illustrates an installation of two different and optional DeviceEmulation Modules, which impart to the apparatus additional capabilitiessuch as facsimile machine emulation.

FIG. 12 illustrates an embodiment specially adapted for "formshandling"--that is, the computer assisted completion of forms, wherein aform (or application) is being inserted between an image forming elementand a reflective back-layer.

FIG. 13 is a view of a "blank form" as seen through the image formingmember.

FIG. 14 illustrates an image forming member bearing informationpertinent to completion of the "blank form" generated on it.

FIG. 15 depicts the information appearing in FIG. 14 superimposed on the"blank form".

FIG. 16 shows a "completed form" that was created by the apparatus of anembodiment of the present invention.

FIG. 17a shows a module insertable into the apparatus 1 for reading adata storage medium comprising a first format.

FIG. 17b shows a module insertable into the apparatus 1 for reading adata storage medium comprising a second format.

FIG. 17c shows a module insertable into the apparatus 1 for reading adata storage medium comprising a third format.

FIG. 17d shows a copier-print apparatus 1 on top of a photocopier. Theapparatus 1 has a display console 100, and a receiving means 177 forreceiving the modules shown in FIGS. 17a-c.

FIG. 17e shows a copier-print apparatus with receiving means 15 forreceiving user-installable option modules 16a and 16b, which are used toextend the functional capabilities of the apparatus.

FIG. 18a shows a copier-print apparatus capable of using data from avariety of different, and mutually incompatible computer platforms.

FIG. 18b shows the insertion of two modules into the copier-printapparatus.

FIGS. 19(a)-19(g) illustrate an example of a process that might beinvolved in creating a hard copy of a disk file using a tablet inaccordance with the present invention.

FIGS. 20(a)-(c) illustrate an example of the versatility afforded by anembodiment of the present invention.

FIG. 21 illustrates an example of the use of a tablet in printingdigital photographs taken by an electronic still camera.

FIG. 22 illustrates an example of the incorporation of a compatibilityassurance card reader into a tablet.

FIG. 23 illustrates an example of a feature referred to as"Pseudo-Collation".

FIG. 24 illustrates an example of a hardware configuration in oneembodiment of the present invention.

FIG. 25 illustrates an example of the application of MicroChannel Busarchitecture by a tablet so that multiple masters can be effectivelysupported.

FIG. 26 illustrates an example of a process for obtaining hard copy witha tablet from data created by a software package, that is saved on amedium such as a diskette.

FIGS. 27 and 28 illustrate an example of an alternate method of gettingdata created by an applications program into a tablet.

FIG. 29 illustrates an example of how a device controller module and afile interpreter module of a tablet might be "borrowed" by aconventional printer.

FIG. 30a shows a portable computer capable of producing hard copy usinga conventional photocopier from applications software run thereon.

FIG. 30b shows a switch provided on the portable computer for enablinginformation to be printed using either a conventional photocopier, or astandard printer attached to the computer.

FIG. 30c shows the display screen of the portable computer, withphotosensors and light gasketing.

FIGS. 31 and 32 illustrate an embodiment of a portable computerconstructed specifically to permit data to be "printed" by using adisplay screen with a copier.

FIG. 33 illustrates an example of a system for optimizing the angularorientation of a hard copy image generated by an image display screen ofa tablet.

FIG. 34 illustrates an example of an operator's console screen that maybe provided on the back of a tablet to visually illustrate a calculatedskew angle.

FIG. 35 illustrates an example of an alternative display in which a skewangle Ω is depicted as an arc of a circle.

FIG. 36a shows a set of buttons used to assist in producing hard copy ona copier.

FIG. 36b shows a hard copy page with an improperly positioned image. Thedot D is below the top margin of the printed sheet, so the tablet mustbe shifted upward.

FIG. 36c shows a hard copy page that has been correctly imaged by thecopier, with alignment dot D at the upper margin of the printed sheet.

FIG. 36d shows a tablet being shifted in a direction to form a correctlyaligned image by a copier.

FIG. 37 illustrates an example of an alternative system for optimizingthe rectilinear orientation of an image on a printed page.

FIG. 38 illustrates an example of an alternative system for optimizingthe angular orientation of an image on a printed page.

FIG. 39 illustrates an example of how a fax document might appeardisplayed on an image display screen of a fax tablet.

FIG. 40 illustrates an example of a light gasket of a fax tablet.

FIG. 41a shows a fax document that has been received by the apparatus.Electronic audio information linked to one portion of the document isbeing played back.

FIG. 41b shows the playback of another linked audio message.

FIG. 42a shows a fax tablet equipped with a document scanner forscanning hard copy documents.

FIG. 42b shows a cutaway view of the fax tablet to reveal the scanningassembly.

FIG. 42c shows the portion of the scanner assembly that processeselectronic data storage media associated with hard copy documents.

FIG. 42d shows the portion of the scanner assembly that processes theoptical component of hard copy documents.

FIG. 43a shows examples of systems the apparatus may communicate with.

FIG. 43b shows the apparatus being used with a copier.

FIG. 43c shows a hard copy document produced by the apparatus with thecopier.

FIG. 44a shows the front face of a hard copy document produced by theapparatus.

FIG. 44b shows the back face of a hard copy document produced by theapparatus.

FIG. 44c shows the hard copy document of FIGS. 44a and 44b beingprocessed by the scanner assembly of the apparatus.

FIG. 44d shows the front face of the hard copy document after it hasbeen processed by the apparatus.

FIG. 44e shows the back face of the hard copy document after it has beenprocessed by the apparatus, with an electronic data storage mediumapplied thereto.

FIG. 45a shows the front face of a hard copy document, which already hasan electronic data storage medium.

FIG. 45b shows the back face of a hard copy document which already hasan electronic data storage medium.

FIG. 45c shows the apparatus processing a hard copy document whichalready has an electronic data storage medium.

FIG. 45d shows an image on the display screen of the apparatus, whichcorresponds to the image on the front face of the hard copy documentprocessed by the apparatus.

FIG. 45e shows access of one piece of linked electronic referenceinformation.

FIG. 45f shows access of another piece of linked electronic referenceinformation.

FIG. 46a shows a document to which audio messages may be added.

FIG. 46b shows defining a highlight box enclosing a portion of thedocument to which an audio passage may be added.

FIG. 46c shows a highlight box enclosing a portion of the document towhich an audio passage may be added.

FIG. 46d shows a completed highlight box.

FIG. 47a shows a highlight box that is ready to be linked to an audiopassage.

FIG. 47b shows buttons used in connection with audio passages.

FIG. 48 illustrates an example of a communications protocol for faxtransmission.

FIG. 49a shows a document having frames and numerals to indicate thelinked portions.

FIG. 49b shows a document having numerals delimiting the linkedportions.

FIG. 50 illustrates an example of a voice capable desktop fax unitcontaining an image display screen that is not touch sensitive.

FIG. 51 illustrates an example of a cradle into which a fax tablet maybe placed.

FIG. 52 illustrates an example of a cradle equipped with a video camerawhich enables the fax tablet docked in the cradle to send, receive anddisplay video images.

FIG. 53 illustrates the use of coordinates to identify positions of linkregions contained in a document having electronically linked electronicreference information associated therewith.

FIG. 54a illustrates a predetermined attribute of a document image inthe form of numerals delimiting linked portions of the document.

FIG. 54b illustrates the step of electronically scanning the electronicrepresentation of the document image to locate the linked portion of thedocument.

FIG. 54c illustrates the recording of the positions in the documentwhere the predetermined attribute has been detected through scanning.

FIG. 54d illustrates the identified link regions in a document based onthe portions of the document image found to have the predeterminedattribute.

FIG. 55 illustrates the manual inscription of graphical identifiers in adocument to designate portions of the document to be linked toelectronic reference information and the resulting document with linkregions where the graphical identifiers were manually placed.

FIGS. 56(a) and (b) illustrate the concept of a remote link, and ways ofstoring this information in an electronic data storage medium capable ofbeing included on a printed document.

FIG. 57 illustrates the use of a non-standard facilities frame to modifya facsimile transmission for transmitting linked electronic referenceinformation.

FIGS. 58(a-d) illustrate the communications scheme of FIG. 48 in greaterdetail.

FIG. 59 illustrates the benefit of synchronizing the formatting of thehard copy document with the storage of information to an EDSM (5117) andthe potential result if synchronization is not utilized (5113).

FIGS. 60(a) and (b) illustrate the organization of linked electronicreference information for convenient and independent accessibility.

FIGS. 61(a) and (b) illustrate an apparatus for playing documents havinglinked electronic reference information.

FIG. 62 illustrates the dynamic interaction between electronic referenceinformation and printed information appearing on a page for documentsretaining electronic links to electronic reference information.

FIG. 63 shows a hardcopy document that will be transmitted by aconventional facsimile machine using a standard fax transmissionprotocol (i.e. CCITT Group 3) as a monochrome bi-level bitmap.

FIG. 64 shows the hardcopy document of FIG. 63 after graphicalidentifiers have been added by hand to appropriate portions. The pageshown in FIG. 64 represents the first of a two page document to betransmitted to the receiving apparatus of the present invention.

FIG. 65 shows the second of two pages to be transmitted. The delimitedportion of the page shown in FIG. 65 is also to be electronically linkedto electronic reference information comprising audio signals.

FIG. 66 is the electronic rendition of the hardcopy page in FIG. 64, onan image display device of the receiving apparatus of the presentinvention. Portions of the displayed page are in color, even thoughthose very same portions were transmitted in black and white by thesending station.

FIG. 67 shows a window containing a document portion that iselectronically linked to another document portion. It is alsoelectronically linked to electronic reference information in the form ofan audio passage.

FIG. 68 illustrates how data associated with each pixel of a receiveddocument image is translated from a first form to a second form, topixel-wise cause the image attributes to be re-mapped.

FIG. 69 shows pixel-wise image data translation applied to collectionsof pixels.

FIG. 70 shows some grammatical examples of one syntactical flavor of adocument image processing language used by the present invention.

FIG. 71 shows the electronic erasure of the pixels that make up thegraphical identifiers graphically added to the document image.

FIG. 72 shows examples of how optically encoded data may be specified ina transmitted document.

FIG. 73 shows a handwritten hardcopy document containing portions to beelectronically linked to electronic reference information, being fedinto the scanner of a conventional facsimile apparatus for transmission.

FIG. 74 shows one embodiment of a fall-back contingency for providingbackwards compatibility between different versions of the presentinvention.

FIG. 75 shows another embodiment of a fall-back contingency forproviding backwards compatibility between different versions of thepresent invention.

FIG. 76 shows some of the choices offered by the voice menu of thereceiving apparatus, for setting up the exchange of electronic referenceinformation between sending and receiving stations.

FIG. 77 shows some examples of the use of a telephone keypad to enteralphanumeric information.

FIG. 78 shows the handwritten hardcopy document 10010 of FIG. 73 afterit has been received by the receiving apparatus.

FIG. 79 shows a receiving apparatus constructed in accordance with thepresent invention establishing contact with a voice telephone number.

FIG. 80 shows a fax document received from a conventional fax machinethat has a link to electronic reference information permitting analphanumeric paging unit to be automatically contacted.

FIG. 81 shows the entry of a text message that will be sent to analphanumeric paging unit.

FIG. 82 shows a handwritten letter that will contain a link to a Website when it is received by the apparatus of the present invention froma conventional fax machine.

FIG. 83 shows how a telephone keypad may be used to communicateinformation such as Web addresses.

FIG. 84 shows how a received fax document can be used to contact a Website automatically.

FIG. 85 shows how a document received from a conventional fax machinemay be used to contact a fax-on-demand server automatically.

FIG. 86 shows a document received from a fax-on-demand service, and theWeb page which can be retrieved from it.

FIG. 87 shows a wireless connection being made using "Dual ComponentDynamic PassCode Protection."

FIG. 88 shows one method of obtaining the Co-PIN Number within a SecureEnvelope.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The following detailed description is of the best presently contemplatedmode of carrying out the invention. This description is not to be takenin a limiting sense, but is made merely for the purpose of illustratingthe general principles of the invention. The scope of the invention isbest defined by the appended claims.

The general form of the invention comprises a device capable ofgenerating an image derived from an electrical representation of animage. The device possessing the capacity to be mounted or otherwiseplaced in such relationship relative to a copier that said copier canreproduce the image onto a substrate such as paper.

The electrical signals defining the image may originate from a computer,magnetic storage device, optical storage device, or from any kind ofsource or electronic apparatus capable of generating, manipulating,storing, or conveying electrical signals representing displayableinformation. The internal components and the manner of operation ofthese devices are well known in the art and, in the interest ofsimplifying the present disclosure, will not be discussed in the presentdisclosure.

Embodiments of the invention, in a sense, can be likened to "electronicpaper" since it is usually a sheet of paper, bearing an image on it,that is placed on a copier for the purpose of being duplicated.Embodiments of the present invention, however, permit images to beeasily altered or manipulated as they are of electrical origin. Thefunction of the copier is to transform the "electronic paper" image intoone on "real paper", or one on other suitable substrates.

As those skilled in the art of electronic information display are aware,there are numerous means by which an image, emanating from an electricalsignal representative of that image, may be fashioned. Numerousstrategies embraced by that technological family may be applicable tothe reduction to practice of the present invention. Examples ofpotential image forming systems are: liquid crystal devices, gas plasmadevices, fluorescent displays, cathode ray tubes, electrophoreticdisplays, and filed emission displays. The particular image formingtechnology used to implement the invention will influence the additionalcomponents needed to enable the image forming element to function asintended; impact the resolution, contrast, and quality of the printedoutput; and affect the overall configuration of the apparatus. Forinstance, electrophoretic displays would not require back-lighting. Theaforementioned considerations are appreciable to those skilled in theart.

Depicted in FIG. 1 is a representation of a known copying apparatus,such as a copier. The known copying apparatus includes a body 4 and acopier window 5. In one embodiment, the invention is in the form of animage-forming tablet. As shown in FIG. 2, the image-forming tablet 1 maybe placed on top of a conventional copier like an ordinary document,with the copier window 5, shown in FIG. 1, immediately beneath thetablet 1. An electrical cable 3 may be used to convey the signalrepresentative of the image to the tablet 1 from a signal source 11. Asdescribed above, the signal source 11 may include a variety of devices,such as a computer, a magnetic storage device, an optical storagedevice, etc.. A light shield 2 may serve to reduce the leakage ofambient light through the copier window 5, and may be constructed of anysuitable, compliant material.

An image forming element of the liquid crystal variety is pictured inFIG. 3. The liquid crystal image forming element includes a liquidcrystal image forming layer 6a, capable of forming an image 60 (in thisparticular example, a set of alphabetical characters assembled into theword "Hello").

FIG. 4 illustrates a reflective back-layer 6b which may be providedadjacent to the liquid crystal image forming layer 6a. For purposes ofclarity, the liquid crystal image forming layer 6a and the reflectiveback-layer 6b are depicted as being spatially separated by a significantdistance in FIG. 4. However, in most embodiments of the invention theselayers would be relatively closely disposed to each other.

The reflective back-layer 6b may be applied directly to the back surfaceof the liquid crystal image forming layer 6a in the form of a coating. Aprimary function of the reflective back-layer 6b is to provide an albedoin the non-imaged areas of the liquid crystal layer 6a that is opticallydistinct from the imaged areas of the liquid crystal layer. Non-imagedareas of the liquid crystal layer 6a are substantially opticallytransparent to radiation of wavelengths within the domain of concern. Itis also possible to substitute for the reflective back-layer 6b anactive, light emitting source, such as an electroluminescent panel.Back-lighting, however, increases the power consumption level.

FIG. 5 depicts the elements of FIG. 4 with the inclusion of analphabetic character string formed on the liquid crystal layer 6a.

FIG. 6 shows a schematic view from beneath the copier window 500 of anordinary copying apparatus 40. The structures illustrated in FIG. 5 areshown positioned on top of the copier window 500, in the manner thatthey would be during the process of duplicating the image 60 formed bythe image forming layer 6a.

The tablet shaped embodiment of the invention described above is fullycapable of being used with low cost "personal" copiers possessing movingcopy boards. An embodiment of the invention specially adapted for usewith such copiers is illustrated in FIG. 7. As depicted in FIG. 7, theimaging element along with its supporting members is in the form of astrip 600. The strip 600 may be held in place by a structural support625, which in turn may be mounted to the body of the copier 4 by afastener 650. Beneath the copier window 5 can be seen that part of thecopier 80 which is sensing the pattern formed by the strip 600 forreproduction onto a substrate. Interposed between the image formingelement 600 and the copier window 5 is a transparent copy board 7 whichis the part of the copying apparatus that undergoes translationalmotion, and is the member upon which any document to be copied isordinarily placed.

FIG. 7 is intended to demonstrate one configuration of the presentinvention that may be used in conjunction with low cost "personal"copiers possessing moving copy boards. It will be recognized by thoseskilled in the art that numerous alternative configurations are possiblefor such use.

FIG. 8 illustrates an embodiment of the invention in which the imageforming element is of the emissive variety; that is, of the type whichemits radiation, e.g., light. Such an image forming element may, forexample, employ a fluorescent display, a gas plasma display, etc. Theimage forming element 65, during the course of operation, may includeone or more portions 65a that are not emitting light and one or moreportions 65b that are emitting light. It is the sum of the non-lightemitting areas 65b that collectively comprises the image to be printed.

In FIG. 8 the non-emitting areas 65a are seen to compose the characterstring "HELLO" against the background of emitting areas 65b. Thephotoconductor of a copier will be discharged by the light emittingportions 65b, to leave behind on the photoconductor a latent imagecharge pattern substantially identical to that defined by regions 65a.Also shown in FIG. 8 is an anti-reflective layer 70. The function ofthis anti-reflective layer is to reduce the amount of light emanatingfrom the copier's internal light source that is back-reflected off ofthe surface of the image forming member of the apparatus of the presentinvention. In practice the anti-reflective layer is more likely to bepresent as a surface coating on the image forming member. Ananti-reflective layer may be included in other embodiments, such asthose employing liquid crystal components.

FIG. 9 is a diagrammatic representation of an arrangement where theelectrical signals defining the image to be printed are transmitted byan infrared beam, rf-carrier, or other "intangible medium" from a source300 to the apparatus of the present invention 1. The point of receptionof the information-bearing signal is shown as 30. The "intangiblemedium" that substitutes for the electrical cable or connector isindicated by 35. The copying apparatus is designated as 4; and thewebbing which shields ambient light from leaking through thetablet/copier interface is identified as 2.

FIG. 11 illustrates the installation of two Device Emulation Modulesinto "Module Bays", 15. In FIG. 11, a first module, 16a, is designed toperform a first function, and a second module, 16b, designed to performa second, different function. An example of a function which a module16a or 16b may perform is that of facsimile machine emulation, asdescribed in further detail below. Whereas two Modules are shown in FIG.11, the apparatus of the present invention is not restricted to thisnumber. Also, while some embodiments contain data storage devices suchas disk drives, these are not represented in FIG. 11.

Some embodiments incorporate into the apparatus the means to readdirectly from data storage media such as magnetic diskettes, magnetictapes, or optical discs. This facility permits text or graphicalinformation to be directly loaded into the apparatus and obviates theneed for connection to another device such as a computer. As someembodiments of the invention may be powered by batteries, certainversions, especially when coupled with the capability to read directlyfrom data storage media are highly self-contained and portable:requiring for operation only a conventional copier and a diskette (forinstance) containing material to be printed.

An embodiment specially adapted for "forms printing" includes amodification to the spacing between image forming element 6a andreflective back-layer 6b (which were discussed earlier in connectionwith FIGS. 4 through 6).

As illustrated in FIG. 12, a blank form 90 is inserted between members6a and 6b, with the front of the blank form facing 6a. The actualspacing between members 6a and 6b may be changeable so as to accommodateforms of different thicknesses; and means may be provided to adjust thespacing to suit the inserted form. Additional means may be provided toalter the relative positions of layers 6a and 6b to facilitate insertionand removal of blank forms, as well as to insure their proper alignmentwithin the tablet.

FIG. 13 demonstrates how blank form 90 would appear viewed through imageforming member 6a, which is the copier's perspective during thereproduction process. FIG. 14 depicts the image forming member, 6a, withalphanumeric information, 61, generated on it. In the illustratedexample, the information comprises two data fields that are pertinent tocompletion of blank form 90: a name, and a street address. Theinformation may be derived from a software package such as a data basemanager, a spreadsheet, or a word processor. FIG. 15 portrays the visualoverlay that results when blank form 90 is situated behind image formingmember 6a, and displayable information is present on member 6a.

Exhibited in FIG. 16 is the printed output of a copier that has beenused in combination with the tablet of an embodiment of the presentinvention operating in "forms printing" mode. It can be seen that theoriginal blank form, 90, has been duplicated and that the data discussedin connection with FIGS. 14 and 15 have been imprinted on the replica;they are indicated by 66. Other information, 67, relevant to thecompletion of blank form 90 has, in a similar manner, likewise beenimprinted. The "filled out form" output by the copier is labeled 99.

In addition to serving as a portable conversion device that enables aconventional copier to function as an electronic printer, high speedprinting matching and even exceeding the capability of dedicatedelectronic printers can be achieved by using the present invention inconcert with high performance copying equipment. Further, other benefitsprovided by such equipment including large volume printing, andelaborate paper handling facilities like collation and stapling can beconveniently tapped.

Not only do embodiments of the present invention permit an ordinarycopier to be utilized as a computer printer, but it also enables theconventional copier to function as a "receiving end" facsimile printer.In one embodiment a facsimile module can be plugged into the tablet toenable fax data transmitted over standard telephone lines to beintercepted, interpreted, and converted into displayable form. Oncepresent on the image forming member, the graphical data can bereproduced by a copier. Further, it should be noted that the facsimileinformation can be directly viewed off of the image forming member, andthus need not be printed unless desired. In this manner the presentinvention provides for both "optical fax" and conventional "hard copyfax", and permits the user to inspect a transmitted document in entiretybefore deciding whether all or only portions are to be printed.Variations on this theme include embodiments possessing documentscanning capabilities, so that data transmission is possible, inaddition to data reception. Other embodiments are equipped with storagemeans so that incoming data can be saved, then printed in a time-shiftedfashion at another moment.

Embodiments of the invention may possess means to detect when a copierhas completed its scan cycle; that is, when an image to be duplicatedhas been successfully captured by the copier. An example of a detectionmeans is a photosensor 10 shown in FIG. 10. Shortly after thephotosensor 10 detects the flash from the copier's source ofillumination, the next image, or page, is formed on the image formingmember. With the next image present, the succeeding copy cycle printsthe next page of information. Multiple copies of each page are handledby counting the number of flashes emanating from the copier; when thecount reaches the number of copies desired for that page, the next pageis formed on the image forming member. The copier must be set to thetotal number of pages that are to be printed.

In practice it is convenient to employ four photodetectors positionedapproximately at the comers of the image forming tablet. After all fourphotodetectors register exposure to the copier's source of illumination,"page flipping" on the image forming member takes place. "Page flipping"does not take place if all of the photodetectors do not register anexposure, since such a situation may be the result of an errorcondition: for example, the page on the image forming member may nothave been completely scanned by the copier. As would be clear toindividuals skilled in the art of circuit design, the above-describedoperation is easily accomplished by simple logic circuitry; which ispreferably incorporated into the tablet of the present invention. Themethod described for controlling "page flipping" applies to copiers withscanning light sources and copiers with flash type light sources withequal facility. Manual "page flipping" is also possible, but requiresthe sustained presence of an operator. Furthermore, it will berecognized that other types of detection means for detecting when acopying operation is being, or has been, performed may be employed(e.g., thermal sensors, current or voltage sensors, etc.).

While the placement of an image forming element of one of the kindsdescribed herein within an otherwise typical, dedicated copying tabletis possible, it has not been the objective here to propose devicemodifications. In contrast, the goal of the present invention is to makeuse of existing conventional, dedicated copying equipment to printimages created or stored in an electronic format that heretofore hasnecessitated the use of an tablet such as a "computer printer".

Examples of Option Modules, which provide device emulation means, arediscussed in greater detail below in connection with FIGS. 17 through32. The following discussion is divided into two parts. First, examplesof some general features and operational characteristics of a tabletfeaturing option modules are presented. Second, a treatment of some ofthe technical considerations involved for reducing this aspect of theinvention to practice is provided.

I. Examples of Some Option Module Characteristic

It is recognized that different computer platforms have differentstrengths. Individuals skilled in the use of more than one type ofmachine may feel inclined to migrate from one piece of equipment toanother in an effort to maximally utilize available resources. This isespecially true at universities and in research settings. One of theproblems encountered by people who employ a multiplicity of platforms intheir work is the need to return to each of the platforms used in orderto produce hard copy. This may mean returning to a number of differentand geographically segregated sites. Another problem is that thesoftware originally used to create each portion of the whole work muststill be present (or available) on each of the platforms used. Whenhardware is shared among many users, it is not uncommon for software tobe erased by people seeking to free-up disk space. This means having tore-install applications software--a potentially time consuming process.

One of the primary objectives of the present invention is to providehard copy of electronic format information arising from diverse origins.Another objective is to provide hard copy of electronic formatinformation arising from diverse origins in a manner that is flexibleand accessible, even to people with non-technical backgrounds. Yetanother objective of the present invention is to accomplish the abovegoals in a fashion that is economical and user-customizable. These andother objectives are facilitated by Option Modules.

Prior to the advent of the present invention, in order to obtain hardcopy of information generated by a particular type of equipment, oneneeded to return to, and to employ, that specific type of equipment. Forinstance, a diagram drawn on an Apple Macintosh computer would typicallynecessitate the use of an Apple Macintosh computer in order for aprintout to be produced. A document written on a Smith Corona dedicatedword processor would require the application of such a unit for printout to be obtained. A photograph taken by a still video camera wouldcustomarily make use of a video-floppy disk printer to make the journeyfrom electrical signals to ink on paper. Because the inventory ofequipment that generates electronic format data is enormous, andcontinuing to grow, the complexity of producing hard copy is alsoincreasing.

The present invention employs an ordinary copier as a source of a printengine. It then builds on this with the support of Option Modules which,by providing device emulation means, transforms an ordinary copier intoa universal print engine. Furthermore, because the user interface hasbeen tailored to people having non-technical backgrounds, informationgenerated on a variety of computer platforms and devices is madeaccessible even to those who do not know how to operate the equipmentthat originally created the information.

Because a modular approach is employed, users can customize a tablet inaccordance with the present invention to suit their particular printingneeds. For example, one who only uses an Apple Macintosh computer caninstall a Macintosh-only option module for printing out Mac files savedon floppy diskettes. Similarly, for those whose computing needs are metexclusively by IBM/DOS machines, only an IBM/DOS option module need beused. University libraries, on the other end of the spectrum, are likelyto have installed more costly multi-platform adapters so that filesoriginating from a range of sources, such as IBM PC's, Macintoshes,Commodore Amiga's, Next Stations, Smith Corona word processors, etc. canall be printed using a library copier. Not only is the modular approachof the present invention functionally adaptable, but it is alsoeconomically flexible. Because different copiers can be used inconjunction with a single tablet, the overall system performance can becustomized by the user. For example, a tablet might be used with a lowcost "personal copier" for ordinary print jobs. The same tablet mightlater be used with a high speed collating copier to print archival datafrom a CD-ROM. And still later, that same tablet might be employed, forexample, with a "blue print" machine to print an architectural blueprint. Thus, the overall system can be tuned to meet specific printingneeds by varying the choice of option module and print engine--thelatter being conveniently and economically furnished by an ordinarycopier. FIGS. 17(d) and (e) illustrate an example of a tablet 1 equippedwith device emulation module receiving means 15, into which optionmodules 16a and 16b may be inserted. Additional receiving means 177 mayalso be supplied in some embodiments to allow data storage devicemodules such as 17a, 17b, or 17c to be installed. In FIGS. 17(a)-(c),the illustrated data storage device module 17a comprises a 3.5" floppydiskette drive; the illustrated data storage device module 17b comprisesan optical storage unit such as a CD-ROM drive; and the illustrated datastorage device module 17c comprises a tape drive.

The illustrated device emulation controller module 16a contains astorage device controller (e.g. an Apple Macintosh 3.5" diskette drivecontroller, an IBM 5.25" diskette drive controller, a Next machine 3.5"drive controller, etc.) to control the data storage device module 17.Depending on the data storage device module 17 installed in the tablet1, different storage device controllers may be used. The deviceemulation controller module 16a may also comprise a "multi-media"storage device controller, capable of controlling a range of storagedevice types, and thus of providing access to a plurality of datastorage formats. If this is the case, then a single device emulationcontroller module 16a would be capable of controlling a plurality ofdata storage device modules 17. The device emulation controller module16a will incorporate circuitry similar to that which may be currentlyfound in diskette drive adapters, tape drive adapters, CD-ROM adapters,etc.

Also illustrated in FIG. 17(e) is a second device emulation module 16bwhich provides the means to translate data from one format into another.The source data format would typically be the "native format" of anapplications software package, such as WordPerfect, Lotus 1-2-3,Paradox, etc.. As used herein, the term "native format" means thedefault data storage format of the applications program that created thedata. For instance, "Publisher's Paintbrush", a popular paint programfor IBM-compatible platforms, has "PCX" as a native storage format.

The applications file interpreter module 16b operates in concert withthe storage device controller present in the device emulation controllermodule 16a and the data storage device module 17. The applications fileinterpreter module 16b enables the tablet 1 to access and then totranslate (or interpret) information stored in different native formatson various data storage media into a form that may be displayed on theimage display screen of the tablet 1. Once in screen displayable form,such information may be printed, as on plain paper hard copy output P bythe copier 4, as illustrated in FIG. 17(d).

Additionally shown in FIGS. 17(d) and (e) is an example of an operator'sconsole 100, which provides a graphical user interface for the apparatusand may be found on the back of the tablet 1. The operator's console 100may include an ATM-style touch-sensitive operator's console screen, sothat intuitive or "natural" operation of the tablet 1 may befacilitated.

FIG. 18(a) illustrates four 3.5" floppy diskettes 18a-18d, generated byfour different computer platforms, that may be inserted into the datastorage device module 17a of the tablet 1. The illustrated 3.5" floppydiskettes 18a-18d contain data saved in a variety of different andmutually incompatible formats. For example, the data on the diskette 18awas created by an IBM/DOS machine, the data on the diskette 18b wascreated by an Apple Macintosh, the data on the diskette 18c was createdby a Next machine, and the data on the diskette 18d was created by aCommodore Amiga.

In the illustrated example, the device emulation controller module 16ais preferably a multi-format storage device controller. This eliminatesthe need to plug in a different, dedicated controller module prior tothe insertion of each different type of diskette. If, on the other hand,it is known that the tablet 1 will only be used to reproduce documentsstored in, for example, IBM/DOS format, the device emulation controllermodule 16a may be equipped to read only IBM/DOS format diskettes.

Techniques for enabling a device emulation controller module 16a tooperate as a multi-format storage device controller are well known tothose skilled in the art. That is, the capability of reading diskettesof more than one format using a single disk drive are well known. As anexample, disk drive controllers that permit both Apple Macintosh and IBMPC diskettes to be read have been available for some time. "CentralPoint Software, Inc." of Beaverton, Oreg. has long been marketing ahardware adapter board for IBM compatible computers called the "OptionBoard" (and after 1988, the "Deluxe Option Board") that allows IBMcompatible computers to read Apple Macintosh diskettes. There are other,similar diskette drive adapter manufacturers. In a somewhat similarfashion, Apple Macintoshes may also read IBM/DOS diskettes. Thus, theproduction of a device emulation controller module 16a which wouldenable the tablet 1 to read diskettes from a variety of differentplatforms, as shown by way of example in FIGS. 18(a) and (b), would bewell within the purview of the skilled artisan.

The file interpreter module 16b performs the function of converting datafrom various source formats into the screen control codes used togenerate displayable images on the image display screen 6a of thetablet 1. Electronic format data files are generally comprised of twocomponents. First, there is the raw data itself. Second, there is theset of formatting instructions that describe how the raw data is to bemade use of. Accessing a data storage medium, such as a magneticdiskette, is only half of the requirement for being able to print nativeformat files saved on native storage media. Interpreting, ortranslating, the accessed native format file is the other half.

As a familiar example, consider a word processor data file. The file maybe composed of the words that constitute the document. The file may alsobe composed of the formatting commands, or control codes, that describehow the words are to be arranged on each page of the document, howtypefaces should appear and in what sizes, whether some words areunderlined, and so forth. Similar concepts apply to other types of datafiles, whether they originate from spread sheet programs, data basemanagers, paint or draw programs, etc.

The file interpreter module 16b provides the means to make sense of thedata files. That is, the file interpreter module 16b provides the meansto apply the formatting rules of different software applicationsprograms to the raw data contained in the data files created by suchapplications. The result of this action is the creation, in screendisplayable form, of information formatted as it was intended to appearby the original author of the data file. The displayed information maysubsequently be copied by a copier in accordance with the generalprinciples of the present invention, as previously described.

FIGS. 19(a) through 19(g) illustrate an example of the process thatmight be involved in creating a hard copy of a disk file using a tabletin accordance with the present invention. The illustrated process isextremely simple, and may be performed by people with non-technicalbackgrounds, including people who do not know how to use a computer.

FIGS. 19(a) through 19(g) illustrate the images that might appear on theoperator's console screen 100 to guide a user in obtaining a printout P.The operator's console screen 100 depicted in FIGS. 19(a)-19(g) is atouch sensitive screen, similar to screens used in some automatic tellermachines. The images displayed on the operator's console screen 100 leadthe user through the process of selecting and printing files selectedfrom a disk 18a inserted into the data storage device module 17a of thetablet 1.

The process begins with the insertion of a disk, such as an IBM/DOSformat floppy disk, into a data storage device module 17a (see, forexample, FIG. 18(a)). Then, as shown in FIG. 19(a), the operator'sconsole screen 100 presents a listing 210 of the files detected on theinserted disk. Platform identification information 212 may also appearat the top of the operator's console screen 100.

As shown in FIG. 19(b), the user may select the files to be printed bytouching the touch sensitive operator's console screen 100. The selectedfiles appear in highlight boxes 215. To de-select a chosen file, theuser might simply touch the selection again. The selection would thentoggle off. When the user has completed making selections, the "Done"screen-button 214 may be pressed. If the user is unsure about theprocedure, or requires assistance, a "HELP" screen button 247 may beprovided.

As shown in FIG. 19(c), a list of the selected files 217 may then bedisplayed, together with a list of applications software origins 219.The user may then indicate what software package was utilized to createthe selected files, with the designated software origins being indicatedin the form of numerals 222 alongside each selected file (see FIG.19(a)). As illustrated in FIG. 19(d), for example, the files"Organic3.LEC" and "Organic2.LEC" represent files created usingWordPerfect software. The origin number of these files would thereforebe "6. WordPerfect". The file "Organic1.LEC" represents a file createdusing "Ventura Publisher" software. The origin number of this file wouldtherefore be "8. Ventura Publisher". When the user has completedindicating what software package was utilized to create the selectedfiles, the "Done" screen-button 214 may be pressed. (To return to aprevious screen menu, the user may press the "Go Back" screen button249.)

As shown in FIG. 19(e), a variety of print options 224 may then bepresented as electives. These include, for example: multiple copies, afeature called "pseudo-collation" (described later in connection withFIG. 23), separator sheets, title sheets, print halt options, printeradjustment options, etc.

As shown in FIG. 19(f), the user may be presented with a printpre-commencement advisory screen 225. The advisory screen informs theuser of the total number of pages that will be generated, so that thecopier may be adjusted accordingly. As shown in FIG. 19(f), for example,the total page count is indicated as being 87. This page count includesall page-consuming special features, such as separator sheets. Asindicated in the advisory screen 225, printing may be initiated bydepressing a copy start button on the copier.

As shown in FIG. 19(g), the user may monitor print status during theprinting process. The "page number box" 232 displays the page currentlybeing displayed on the image-forming screen of the tablet. This allowsthe user to know the status of the print job while the print job istaking place. To suspend the printing process, the copy cycle may beinterrupted at the copier. Since, in a preferred embodiment, the tabletfollows the copier by "watching out" for copy execution via sensors,halting the copier will also place the tablet into suspended operation.Alternatively, a "Stop" screen button 267 may be provided to suspendprinting. Resumption of printing may be initiated by depressing a"Continue" screen button 269. FIG. 19(g) also shows a listing of filesremaining in the print queue 234.

FIGS. 20(a)-(c) illustrate an example of the versatility afforded by anembodiment of the present invention. FIG. 20(a) shows a diskette 18athat may be inserted into a data storage device module 17a of atablet 1. The diskette contains a word processor document that wascreated on an IBM/DOS machine. The hard copy output P1 and P2 producedby the copier 4 comprises text material.

As shown in FIG. 20(b), another diskette 18b may subsequently beinserted into the same data storage device module 17a of the tablet 1.The second diskette 18b may contain drawings that were made on anothercomputer platform, such as an Apple Macintosh, using a program such as"Beaker", which is not available to an IBM/DOS platform. The hard copyoutput P3 and P4 produced by the copier 4 comprises molecular diagrams.

As shown in FIG. 20(c), an ordinary textbook 73 may be copied (printed)alongside any of the above-mentioned electronic format information. Thehard copy output P5 and P6 produced by the copier 4 comprises pagescopied directly from the textbook 73.

All of the above printing of hard copies P1-P6 may be accomplished atthe same physical site, using the same equipment, and in a "continuousflow" that is both intuitive and user-friendly. Moreover, the presentinvention brings these capabilities to ordinary users because all of therequired equipment is relatively affordable. The copier 4 may be alow-cost "personal copier". Still further, the present inventionprovides room for growth, as more expensive, higher performance copyingequipment can be substituted at a future time. The present invention istherefore highly adaptable to one's needs and resources.

FIG. 21 illustrates an example of the use of a tablet 1 in printingdigital photographs taken by an electronic still camera 110. Theillustrated electronic still camera 110 may store pictures on 2-inchvideo floppy disks 18V in a standard format defined by the "ElectronicStill Camera Standardization Committee". The data storage device module17V may comprise a video floppy disk reader installed in the tablet 1 toenable video floppy disks 18V to be read and printed. The hard copyoutput P7 comprises a paper print specimen of a photograph taken by thecamera 110. The ability to create rapid photographic proofs in thismanner may be desirable to organizations such as news bureaus.

FIG. 22 illustrates an example of the incorporation of a compatibilityassurance card reader 130 into the tablet 1. One purpose of acompatibility assurance card 120 is to insure that the tablet 1 may beused to print files created by various software packages. Anotherpurpose is to accommodate changes in data storage formats by softwaremanufacturers.

The compatibility assurance card 120 may comprise a credit-card sizedelement provided by software manufacturers to licensed owners of theirproducts. For example, the manufacturer of a word processing softwarepackage might include a compatibility assurance card 120 in the package.The compatibility assurance card 120 may contain information on the datastorage convention utilized by the latest release of the software. Eachof the formatting and control characters used by the software programwhen writing data to disk files would be available on the card 120. Byinserting the card 120 into the card reader 130, the tablet 1 maydownload the storage convention information from the card into thememory of the tablet 1. This information may then be used in place of,or as a supplement to, the information contained in an installed fileinterpreter module 16b.

A person who, for instance, utilizes a plurality of software packages inthe course of conducting business might be provided with a plurality ofcompatibility assurance cards. In this fashion, such an individual wouldbe assured of being able to print out data at suitably equipped copiersat field locations, at libraries, post offices, drug stores, etc. It isanticipated that most people would require three or fewer compatibilityassurance cards: for example, one for a word processor, one for a spreadsheet program, and one for a data base manager.

Compatibility assurance cards 120 may comprise plastic cards similar tocredit cards, having a magnetic recording layer in which informationsuch as symbol tables for data file translation may be retained.Compatibility assurance cards 120 may also comprise "smart cards",optical memory cards, or any other type of medium that is cost effectiveand that is capable of holding the required format translationinstructions.

FIG. 23 illustrates an example of a feature referred to as"Pseudo-Collation". "Pseudo-Collation" is a technique that permitslow-end copiers that are not equipped with collating capability tosimulate the benefits of collation. When a tablet in accordance with thepresent invention is used to print multiple copies of a single document,the tablet may handle the task as follows: the first document page isdisplayed and printed, the second document page is displayed andprinted, the third document page is displayed and printed, and so onuntil the final document page is displayed and printed. The routine isthen repeated a number of times, depending upon the number of copiesdesired.

Between displaying the final document page of one copy set and the firstdocument page of the next copy set, a separator page may be displayed tothe copier by the tablet. In FIG. 23, for example, the separator pagesare illustrated as pages 167a, 167b and 167c. Also shown in FIG. 23 is apage Px preceding the separator page 167a and a page Px between theseparator pages 167a and 167b. The pages Px denote complete documentcopy sets. In the illustrated example the borders of the separator pages167a-167c are darkened. When a stack of printed pages Pm is fanned, thedark borders of the separator pages 167a and 167b facilitate location ofthe document copy sets preceding and following the separator pages. Thisallows low-end, non-collating copiers to produce multiple copy sets ofdocuments in a fashion that approximates true collation.

II. Examples of Some Technical Considerations

Option Modules, which provide device emulation means, supply a mechanismby which users may customize, expand, upgrade, and modify a tablet inaccordance with the present invention. For example, a print spoolermight be installed to more effectively handle large volume print needs.Option Modules further provide the capability to directly access nativeformat data files on the original medium of the devices that createdthem. Option Modules enable printout to be obtained using only a copierand a diskette (for example) containing data to be printed. A connectionto a computer becomes unnecessary. Using a native format file reader asan illustrative example, examples of some Option Module implementationsare discussed below.

FIG. 24 illustrates an example of a hardware configuration in oneembodiment of the present invention. In the illustrated embodiment, allof the components comprise part of the tablet, with the exception of thedevice emulation controller module 16a, the file interpreter module 16b,and the data storage device module 17, all of which are Option Modules.

In the illustrated example, the device emulation controller module 16acomprises a "multimedia" controller capable of controlling more than onetype of storage device. The device emulation controller module 16a may,for instance, be capable of controlling an Apple Macintosh 3.5" diskettedrive as well as an IBM/DOS format 3.5" diskette drive. The method ofaccomplishing this is known in the art. Several computer peripheralmanufacturers have long been marketing products that enable a single3.5" diskette drive to read and write in the storage format of more thana single computer platform. Examples of such manufacturers are numerous,but include Central Point Software of Beaverton, Oregon, and AppleComputer, Inc. of Cupertino, Calif. (for use by Apple). The deviceemulation controller module 16a is connected to the storage devicemodule 17, and is also connected via the I/O bus 385 to the tablet 1.

The file interpreter module 16b contains a native format fileinterpreter. The file interpreter module 16b is also plugged into theI/O bus 385 by virtue of being installed into the module receiving bay15. The file interpreter module 16b performs the function of convertingnative format files into screen displayable form. The file interpretermodule 16b may comprise a ROM. The ROM may include translation tablesand interpreter instructions for translating data, appearing in thenative format of various applications software packages, into a singletarget format. The target format will ordinarily be the screen controlcodes 305 used by the video controller 350 to generate images on theimage display screen 6a of the tablet 1.

As appreciated by persons skilled in the art, the screen control codes305 may comprise a graphics language, such as PostScript. (The screencontrol codes 305 may also comprise a bitmap, rather than a truelanguage.) If the screen control language is PostScript, thentranslation may not even be required in some instances, since certainsoftware programs permit data to be saved in this format as an outputoption. Techniques for translating data from one storage format intoanother are known in the art. Many software applications packagesprovide what are referred to as "Import" and "Export" facilities. Thesefacilities are essentially software translators. Additionally, there areseveral software manufacturers that specifically market formatconversion programs. Examples of such manufacturers are Inset Systems,which produces "HiJaak", and SCC, which produces "Software Bridge". Themethods of data format conversion are therefore well established in theart.

Returning to FIG. 24, a secondary processor 376 is shown having accessto both the I/O bus 385 and the common bus 383. As is standard practicewhen a single processor has dual bus access, buffers 353 are provided toenable bus selection. The buffers 353 are similarly present for theprimary processor 375, which communicates with both the common bus 383as well as primary bus 381. A memory block 364 is present on the I/O bus385. A subset of the memory block 364 is reserved for non-volatilememory 365. When a storage medium, such as a diskette, is placed intothe data storage device module 17, the device emulation controllermodule 16a, under the control of the secondary processor 376, may readdata from the medium. The acquired data may be placed into the memory364. Using format translation rules provided by the file interpretermodule 16b, the processor 376 may translate the native format data readfrom the diskette into the screen control codes (language) used by thevideo controller 350. The translated screen control codes may be placedinto the memory 362 by the secondary processor 376. The primaryprocessor 375 moves the translated screen control codes from the memory362 into the local memory 360. The video controller 350 is able toaccess the screen codes from the memory 360 in order to generate images60 on the image display screen 6a. While the primary processor 375 ismoving data from the common memory 362 to the system memory 360, thesecondary processor 376 may continue retrieving and translatingadditional native format data from the data storage device module 17.

Also shown in FIG. 24 is a ROM unit 367, which contains instruction codefor controlling basic system operation. An EPROM may be substituted forthe ROM unit 367 in some embodiments, so that new system code can beincorporated into the tablet simply by downloading the new code from adiskette into the EPROM. A non-volatile memory block 365 may be used toretain translation instructions which are supplemental to theinstructions found in the file interpreter module 16b. The supplementalinstructions may be downloaded from data storage media such asdiskettes, which may also be accessed using the data storage devicemodule 17. The memory units (e.g. 362, 364, 365) may be used as shadowRAM for instruction code or translation code saved in ROM (e.g. 367, thefile interpreter module 16b). This may offer an improvement in overallperformance of the tablet.

FIG. 25 illustrates an example of the application of MicroChannel Busarchitecture by the tablet so that multiple masters can be effectivelysupported. In this configuration either Option Module 1 or Option Module2 can take control of the overall system from the default system 390(master). Either Option Module 1 or Option Module 2 can control thefunction of the tablet. For example, an option module might be installedto operate the tablet as a "printer" that serves a computer network. Inthe illustrated example, Option Module 1 corresponds to the deviceemulation controller module 16a and Option Module 2 corresponds to thefile interpreter module 16b. However, the present invention does notrequire that the Option Modules correspond identically to either thedevice emulation controller 16a or the file interpreter 16b.

FIG. 26 illustrates an example of a process for obtaining hard copy witha tablet from data created by a software package, that is saved on amedium such as a diskette. In FIG. 26, element 301 represents a storagemedium, such as a diskette, that contains a file that was created by anapplications program. The data is in the "native" format of theapplications program that created it. As used herein, the term "native"means the format customarily employed by the program for the storage ofinformation.

Step 302 denotes the process of accessing a file on the native storagemedium. This involves reading the storage medium (which may be, forexample, a floppy diskette). Step 302 is accomplished by having astorage device controller, appropriate for the storage format ofinterest, control the storage device so as to be able to access thestorage medium. For example, step 302 might entail employing hardwaresimilar to an IBM/DOS floppy diskette drive "adapter" to control a 3.5"diskette drive, so that data stored on a 3.5" IBM/DOS format diskettemay be read. The diskette drive "adapter" of this example may beincluded in the device emulation controller module 16a, and the 3.5"diskette drive may be furnished by the data storage device module 17. Asmentioned earlier, the device controller module 16a may be a"multi-media" storage device controller capable of providing access tothe storage media of more than one type of computer platform.Multi-media controllers are familiar in the art, and the manner of theirapplication and construction would be known to skilled artisans.

In FIG. 26, element 303 represents data that has been read into thememory of the tablet. After being read into the tablet, the data mayoptionally be placed into another storage medium such as, for example, afixed disk drive connected to the tablet. The data 303 is still in itsnative format. That is, for example, if the data 303 started out as aWordPerfect document file, it is still in WordPerfect file format.

Step 304 represents the step of translating the native format data intoa form that can be displayed on the image forming screen of the tablet.The element 305 represents the screen control codes that the nativeformat data is translated into in step 304. The screen control codes 305may comprise the instructions of a display language, such as PostScript.The screen control codes 305 may also comprise the bitmap of the imagethat is to be displayed. In the latter case the screen control codeswould not constitute a true language. The screen control codes 305 mayalso comprise the linguistic elements of numerous other techniquescurrently available for representing graphical information on displaydevices, including novel display languages fashioned specifically foruse by the tablet.

The process of translating native format data into the screen controlcodes, or language, used by the tablet, would depend on the choice ofscreen language adapted by the particular embodiment of the tablet. Theprinciples behind translation, however, are well known to softwareengineers. "Import" and "Export" functions provided by many softwarepackages perform translation between different file formats. Forexample, users of Microsoft Word may "Import" documents created byWordPerfect, and edit them using Microsoft Word instead. Similarly, mostgraphics programs can read and write "BMP", "PCX", and "TIFF" fileformats, regardless of the native format of the program itself. Inaddition to the "Import" and "Export" features built into mostapplications programs, are the software packages that specificallyprovide the service of format conversion. Examples are "HiJaak" by InsetSystems, and "Software Bridge" by SCC. The principles governing formatconversion are thus familiar in the art.

Step 306 of FIG. 26 represents the step of image generation by the videohardware 350 of the tablet. An image 60 is displayed on the imagedisplay screen of the tablet. The process by which an image may begenerated from data and displayed by video hardware is ubiquitously wellknown. The graphics adapters of all computers perform this function.

FIGS. 27 and 28 illustrate examples of alternate methods of getting datacreated by an applications program into the tablet. As illustrated inFIG. 27, a program 410 is run by a computer 408. As a result of runningthe program 410 on the computer 408, data 412 is created. The step 409denotes the act of sending data to a printer 415, by way of executing aprint command from within the program 410. The data 412, however, doesnot reach the printer 415, because it is intercepted by a TSR (Terminatebut Stay Resident) Program 414a. The TSR program 414a redirects theprint stream to a storage medium, such as a floppy diskette 401a. Thedata file recorded on the storage medium 401a may contain printercontrol language data, since the data stream was originally intended fora printer. The step 302 involves accessing the storage medium 401a, asin FIG. 26. When the storage medium 401a is read, the printer controlcodes are read into the memory of the tablet. The element 405 representsprinter control codes in the RAM of the tablet. The step 304 involvestranslation of the printer control codes into the screen control codes305. As in FIG. 26 screen image generation 306 causes the data to bedisplayed as an image 60 on the image display screen of the tablet.

FIG. 28 shows another technique for getting data into the tablet. As inFIG. 27, a computer 408 is shown running an applications program 410(for instance, a spread sheet program) which generates data 412. Theuser executes a print command from within the program 410 to send thedata to a printer 415. Step 409 represents the act of sending data to aprinter from within the applications program. Instead of going to theprinter 415, the data stream is translated by the computer 408 into thescreen control codes 305 of the tablet 1 and then written to a storagemedium. The element 401b represents a storage medium, such as adiskette, containing screen control codes representative of the datathat was queued for printing. The element 414b represents a driver thatemulates a printer driver, installed on the computer platform 408. Priorto executing the print command, the user selects from the printer setupmenu of the applications program 410, "Copier Printer" as the printdevice. The driver 414b is then invoked when the print command is issuedfrom the applications program 410. Rather than sending a stream ofprinter control language codes to the printer 415, the driver 414bwrites screen control codes 305 to a storage medium, such as a magneticdiskette. Step 418 represents writing captured data to a storage mediumsuch as a diskette.

Drivers are commonly used by software. For example, on an IBM/DOSplatform running Microsoft "Windows", most applications programs havesetup menus that permit the selection of any printers or devices thatmight be installed. By selecting a device from such a menu, a driver forcontrolling the selected device is selected behind the scene. When aprint command is executed from within an application, the drivercommunicates the device control codes that operates the device. In thepresent case, a driver may be employed that emulates a printer driverand that sends screen control codes to a storage medium, such as adiskette, instead of sending printer control codes to a printer.

After the screen control codes have been placed onto a diskette, as inelement 401b in FIG. 28, the information may be accessed by reading thediskette, as in step 302. Reading the diskette places the screen controlcodes 305 into the tablet RAM. Step 306 involves generating an image,using the data, onto the image display screen 6a of the tablet 1.Element 60 in FIG. 28 represents the displayed image.

FIG. 29 illustrates an example of how the device controller module 16aand the file interpreter module 16b of the tablet 1 might be "borrowed"by a conventional printer. A printer interface adapter 201 is used toprovide a means for the conventional printers 415a and 415b to acquiresome of the functionality afforded by the option modules 16a and 16b.The printer interface adapter 201 is supplied with option modulereceiving means 155, which are analogous to the module receiving bays 15found on tablet 1. In FIG. 29, the interface unit 201 is not furnishedwith storage device receiving means comparable to 177 of tablet 1.Instead, one 3.5" diskette drive 197 and one 5.25" diskette drive 198are provided in a fixed mount. The dual format disk drives 197 and 198may be of the 42 mm twin mount variety, to conserve space. An operator'sconsole 191 may be found on the top of the unit 201. The operator'sconsole 191 has similar characteristics to the operator's console 100 oftablet 1. The interface unit 201 is connected to a printer 415 by anelectrical cable 411.

The device controller module 16a may supply the means to controldiskette drives 197 and 198. The file interpreter module 16b may supplydata interpretation means, so that the data files read by the diskdrives 197 and 198 may be printed by the printers 415a and 415b. Becausethe file interpreter module 16b contains instructions for generating thescreen control codes 305 of the tablet 1, a second level of translationmay be required in order to properly control the printers 415a and 415b.The instructions for translating screen control codes into, forinstance, HP-PCL (a printer control language) may be supplied by auser-installable ROM which may be installed at the outset, when theinterface unit 201 is first attached to the printer 415b. Alternateembodiments of the file interpreter module 16b may, however, be equippedwith "extended" interpretation capabilities that include instructionsfor translating native format data into either screen control codes, orprinter control codes. When the screen control codes are comprised of alanguage such as Postscript, for example, and the printer controllanguage is also Postscript, translation naturally would not berequired. As would be recognized by persons skilled in the art, theprinciples of operation and construction of the interface unit 201 wouldbe similar to the concepts presented above.

FIGS. 30 through 32 illustrate an embodiment of a portable computerconstructed specifically to permit data to be "printed" by using itsdisplay screen with a copier. As shown in FIGS. 30(a) and 30(c), theportable computer 535 comprises a base unit 535B and a detachable screenunit 535S having a display 56A. Around the perimeter of the displayelement 56A is a light gasket 502 for keeping ambient light from leakingin between the display 56A and the copier window 5. The light gasket 502may comprise any suitable, flexible material (e.g. rubber). At thecorners of the display element 56A are a set of sensors 10 for detectingcopy execution. A set of LED's 517 in the form of arrows point towardsthe screen unit 535S. Whenever print data is being diverted to thescreen unit 535S, the LED's 517 blink to indicate that a data stream isbeing captured by the screen unit.

A spring-loaded switch 518 may be slid to the left or to the right of adetente, center position (see FIG. 30(b)). The "normal" setting 521 ofthe switch 518 allows data queued for printout by the computer 535 to berouted to the parallel or serial ports of the computer. In the "normal"setting 521, the execution of a print command from within anapplications program would cause the data stream intended for printoutto be sent to whichever output port or device has been selected insoftware. This is similar to what conventionally occurs when a printcommand is executed on an ordinary computer. The switch position 523,located to the right of the center, detente position, and having a smallcopier icon associated with it, causes a data stream queued for printoutto be diverted to the screen unit 535S.

FIG. 31 illustrates an example of the data capture process of theportable computer 535. As shown, the portable computer 535 comprises abase unit 535B and a screen unit 535S. The arrows 513 indicate that thescreen unit 535S may be detached from the base unit 535B. The base unit535B includes a system unit 503 for running applications software 410.The software program 410 creates the data 412. In screen capture mode,whenever the computer operator executes a print command from withinapplications program 410 to send the data 412 to the "outside world" 357(e.g. via Lpt 1 or 2), the data stream is instead intercepted by asoftware daemon 514 and sent to a capture unit 555 in the screen unit535S. "Daemons" are a member of a larger class of RAM resident softwareprocesses that "transparently" perform specific tasks in the background.Daemons and TSR's (Terminate but Stay Resident programs) may be invokedby "Hot-Key" sequences, interrupts, or by conditions that may ariseduring the course of normal system operation. The entire length ofobject code required to perform the task for which the daemon isresponsible need not be RAM resident. Rather, there may be a smallerportion of the complete code which, when activated, would load theremainder of the code which would subsequently perform the function.These concepts are familiar to software engineers. An example of a TSR(or daemon) might be a program that handles facsimile reception on acomputer. When the facsimile hardware detects a call on its phone line,the TSR (or daemon) is invoked to service the incoming fax transmission.In the present case, the daemon 514 is invoked by the execution of aprint command, or by an attempt to place data onto a serial or parallelport of the computer 535. The data stream that is queued for output viathe serial or parallel port is instead sent to a capture unit,identified as 555 in FIG. 31. The capture unit 555 is described furtherin connection with FIG. 32.

Data sent to the memory 560 of the capture unit 555 may be in a numberof different possible formats. It is possible to construct the daemon514 to write the actual screen control codes 305 used by the videoadapter into the memory 560. For instance, the daemon 514 may include asoftware driver that emulates a printer driver, but which is used togenerate screen control codes instead of printer control codes. This issimilar to a technique used by some computer facsimile programs tocapture a print stream and to transmit it as a fax document, rather thanto an actual printer. The Fax TSR of such devices includes a driver thatwrites the print data as transmission-ready fax data, rather than as astream of printer control codes used to drive a printer. In the presentcase, where screen control codes are placed into the memory 560 of thescreen unit 535S, the capture unit 555 would not have to perform furthertranslation on the data before it may be interpreted by the videohardware. The capture unit may, in such instances, merely comprise thememory 560. In contrast, in some embodiments it may be preferable tohave the daemon 514 place into the memory 560 print stream data in anintermediate format. For example, it may be desirable to performcompression on the data before it is placed into the memory unit 560.The purpose of writing compressed, intermediate format data into thescreen unit memory may be to permit lengthier documents to be loadedinto the detachable screen unit 535S. This intermediate format datawould then need to be converted into usable screen control codes by thecapture unit. Also shown in FIG. 31 are a video adapter 550 and adisplay screen 56A of the computer 535. The video adapter 550 uses thescreen control codes 305 to generate displayable images on the display56A.

FIG. 32 illustrates an embodiment of the computer 535 in which thecapture unit 555 includes logic for converting intermediate formatcapture data into usable screen control codes. Such may be the casewhere the daemon print stream diverter 514 does not write usable screencontrol codes into the memory 560. The top of FIG. 32 illustrates thesystem unit 503 and some peripheral devices with which it maycommunicate. The peripheral devices include a storage device 517, suchas a fixed disk drive, and a compatibility assurance card reader 130 (anexample of which is described above in connection with FIG. 22). Alsoincluded is an I/O Processor 138, which acts as an intermediary betweenthe peripheral devices 130 and 517 and the rest of the system. Thecapture unit 555 is illustrated as being within the detachable screenunit 535S. In the illustrated configuration, the capture unit 555includes a microprocessor 576, a ROM 516, and a memory block 560. Theapplications software 410 shown in FIG. 31 may, in fact, be run by thesystem unit microprocessor 575. Moreover, the software daemon 514 mayalso be run by microprocessor 575. When daemon 514 re-directs print datato the capture unit 555, it is actually causing the microprocessor 575to place the data into the memory unit 560. Using the instructionscontained in the ROM 516, the capture unit processor 576 translates thecaptured data, that was earlier placed into the memory 560, into theusable screen control codes 305. Also present in FIG. 32 is a block ofnon-volatile memory 565. The non-volatile memory 565 may be used toretain supplements to the instructions stored in the ROM unit 516.Supplemental or update information may be introduced into thenon-volatile memory 565 from a diskette, or it may come from acompatibility assurance card. In the latter case, the compatibilityassurance card would be inserted into, and read by, the CAC reader 130.

The present invention enables detection of the amount of misalignmentbetween a tablet and a copier window. The present invention furtherenables the use of image reorientation techniques, so that an imagedisplayed by a tablet may be reoriented and printed by a copier as ifthe tablet and the copier window were in alignment, even if the tabletis initially misaligned relative to the copier window. Examples ofsystems for optimizing the orientation of a hard copy image generated bya tablet are discussed in greater detail below in connection with FIGS.33 through 38.

FIG. 33 illustrates an example of a system for optimizing the angularorientation of a hard copy image generated by an image display screen ofa tablet. FIG. 33 shows, in schematic form, the image display screen 6of a tablet 1 in position on top of a copier window 5. The relativeposition of the image display screen 6 and the copier window 5 may bedifferent each time the tablet 1 is placed on the copier window 5.Consequently, the image displayed by the tablet 1 may occasionally beskewed or misaligned relative to the copier window 5.

As shown in FIG. 33, the tablet 1 defines a rectangle. Each of the fourcorners of the rectangle may be provided with a corresponding lightdetecting sensor S. Each one of the four sensors S may provideinformation to the tablet's microprocessor.

As further shown in FIG. 33, the angle of skew or misalignment betweenthe tablet 1 and the copier window 5 may be designated by Ω. If there isvirtually no misalignment between the tablet 1 and the copier window 5,then Ω is approximately equal to zero.

In the embodiment illustrated in FIG. 33, the light source of the copierscans across the copier window 5 in an upward direction, from the bottomof the page towards the top of the page (as shown by the upward pointingarrows t₁, t₂ and t₃). Consequently, each of the four sensors S is firstexposed to light from the copier light source at a different point intime. As shown in FIG. 33, for example, light from the copier lightsource will expose sensor S₀ first, sensor S₁ second, sensor S₂ thirdand sensor S₃ fourth. The time intervals between exposure of thedifferent sensors S will be determined by the length L, width W andangular orientation of the tablet 1.

The time at which the first sensor S₀ is exposed to the copier 4 lightsource may be represented by t₀. The elapsed time between exposure ofthe first sensor S₀ and exposure of the second sensor S₁ may berepresented by t₁. Similarly, the elapsed time between exposure of thefirst sensor S₀ and the third sensor S₂ may be represented by t₂, andthe elapsed time between exposure of the first sensor S₀ and the fourthsensor S₃ may be represented by t₃.

The width W and length L of the tablet 1 are known constants that may bestored in the tablet's microprocessor. The angle Ω between the actualangular position of the image and the desired angular position of theimage may be calculated by the microprocessor, based upon theinformation received from the sensors S, in accordance with thefollowing formula:

    Ω=Tan.sup.-1 (L/W)*(t.sub.1 /(t.sub.3 -t.sub.1))

The microprocessor thus uses the information received from the sensors Sto electronically calculate the angle Ω. Once Ω has been determined, thetablet 1 may be physically rotated by the calculated angle Ω to therebymove the image to the desired angular orientation. Alternatively, themicroprocessor may use the information received from the sensors S toelectronically rotate the image by an angle Ω, from the image's presentangular orientation to the image's desired angular orientation. Anynumber of known techniques may be used to electronically rotate theimage by the calculated angle Ω.

FIG. 34 illustrates an example of an operator's console screen 100 thatmay be provided on the back of the tablet 1 to visually illustrate thecalculated angle Ω. The unbroken cross bars in FIG. 34 depict theimage's desired angular orientation. The broken cross bars depict theimage's present angular orientation. The angle between the unbroken andbroken cross bars corresponds to the skew angle Ω.

FIG. 35 illustrates an example of an alternative display in which Ω isdepicted as an arc of a circle.

FIG. 36 illustrates an example of a system for optimizing therectilinear orientation of a hard copy image generated by an imagedisplay screen of a tablet. The illustrated system enables the user todefine one or more points on the printed page, and then move the imageand/or tablet relative to the copier window so that the rectilinearorientation of the image may be optimized.

As shown in FIG. 36(a), four buttons B may be provided on the back ofthe tablet 1. Each one of the four buttons B is associated with acorresponding one of the four edges of the tablet 1. The buttons Bprovide data input to the tablet's microprocessor. Depressing any one ofthe buttons B will cause the tablet 1 to produce an image of a dot D onthe image display screen. The dot image will be located on the edge ofthe image display screen, at a position substantially corresponding tothe location of the depressed button B. The hard copy output P of thecopier will then include a printed dot D at a location on the page Pcorresponding to the depressed button B, as illustrated in FIG. 36(b).Optionally, each button B on the back of the tablet 1 may include asmall LED or other light generating device, so that the button B isilluminated after it is depressed.

If the image is optimally aligned, then the dot D printed on the hardcopy output P will appear substantially adjacent the edge of the printedpage (see FIG. 36(c)). The degree to which the printed dot D is spacedfrom the edge of the printed page P indicates the degree to which theimage must be shifted to correct the rectilinear alignment of the image(see FIG. 36(b)).

Once the distance of the dot D from the edge of the page P has beendetermined, the tablet 1 may be physically shifted by the indicateddistance to thereby move the image to the desired rectilinearorientation (see FIG. 36(d)). Alternatively, the image may be shiftedelectronically from the image's present rectilinear orientation to theimage's desired rectilinear orientation. Any number of known techniquesmay be used to electronically shift the image by the required distance.

FIG. 37 illustrates an example of an alternative system for optimizingthe rectilinear orientation of the image on the printed page. As shownin FIG. 37, the back of the tablet 1 defines a rectangular frameconfigured to receive a printed page P from a copier. The rectangularframe defines four edges. Each of the four edges is provided with acorresponding electronic touch sensitive strip ST. The touch sensitivestrips TS provide information to the tablet's microprocessor.

In the example shown in FIG. 37, the strip TS₁, illustrated on the lefthand edge of the frame is designated as a "FROM" strip. The strip TS₂illustrated on the right hand edge of the frame is designated as a "TO"strip. Similarly, the strip TS₃ illustrated on the lower edge of theframe is designated as a "FROM" strip, while the strip TS₄ illustratedon the upper edge of the frame is designated as a "TO" strip.

In operation, a printed page P from a copier is first placed into theframe on the back of the tablet 1. In the example illustrated in FIG.37, the printed page P contains several lines of printed text. If theuser wishes to shift the image upward, as shown in FIG. 37, then theuser first touches a point on the left hand "FROM" strip TS₁corresponding to the actual location of the first line of text on theprinted page P. This sends an electronic signal to the microprocessorindicating the present position of the printed image. Next, the usertouches a point on the right hand "TO" strip TS₂ corresponding to thedesired location of the first line of text on the printed page P. Thissends an electronic signal to the microprocessor indicating the desiredposition of the printed image. The microprocessor than uses theinformation received from the "FROM" and "TO" strips to electronicallyshift the image upward, from the image's present orientation (asindicated by the "FROM" strip signal) to the image's desired orientation(as indicated by the "TO" strip signal). Pages that are subsequentlyprinted by the copier will provide images that are shifted upward by theindicated amount.

In a similar manner, the top and bottom edge strips may be used toreorient the printed image horizontally. If, for example, the userwishes to shift the image rightward, as shown in FIG. 37, then the userfirst touches a point on the lower edge "FROM" strip TS₃ correspondingto the actual location of the text on the printed page P. Next, the usertouches a point on the upper edge "TO" strip TS₄ corresponding to thedesired location of the text on the printed page P. The microprocessorthan uses the information received from the "FROM" and "TO" strips toelectronically shift the image rightward, from the image's presentorientation (as indicated by the "FROM" strip signal) to the image'sdesired orientation (as indicated by the "TO" strip signal). Pages thatare subsequently printed by the copier will provide images that areshifted rightward by the indicated amount.

It should be appreciated that either strip of the pair of stripsparallel to the length of the tablet, or of the pair of strips parallelto the width of the tablet, may serve as a "FROM" strip or as a "TO"strip. The designation of "FROM" or "TO" is relative to which strip ofthe pair is touched first.

FIG. 38 illustrates an example of an alternative system for optimizingthe angular orientation of the image on the printed page P. The systemfor optimizing angular orientation shown in FIG. 38 is essentiallysimilar to the system for optimizing rectilinear orientation shown inFIG. 37.

As shown in FIG. 38, the back of the tablet 1 defines a rectangularframe configured to receive a printed page P from a copier. Therectangular frame defines four edges. Each of the four edges is providedwith a corresponding electronic touch sensitive strip ST. The touchsensitive strips TS provide information to the tablet's microprocessor.

In the example shown in FIG. 38, the strip TS₁ illustrated on the lefthand edge of the frame is designated as a "FROM" strip. The strip TS₂illustrated on the right hand edge of the frame is designated as a "TO"strip. Similarly, the strip TS₃ illustrated on the lower edge of theframe is designated as a "FROM" strip, while the strip TS₄ illustratedon the upper edge of the frame is designated as a "TO" strip.

In operation, a printed page P from a copier is first placed into theframe on the back of the tablet 1. In the example illustrated in FIG.38, the printed page P contains several lines of printed text. If theuser wishes to rotate the image clockwise by an angle θ as shown in FIG.38, then the user first touches a point on the left hand "FROM" stripTS₁ corresponding to the actual location of the first line of text onthe printed page P. This sends an electronic signal to themicroprocessor indicating the present position of the printed image.Next, the user touches a point on the right hand "TO" strip TS₂corresponding to the desired location of the first line of text on theprinted page P. This sends an electronic signal to the microprocessorindicating the desired position of the printed image.

The width λ of the printed page P is a known constant that may be storedin the microprocessor memory. The vertical distance between the actuallocation of the image and the desired location of the image may becalculated by the microprocessor, based upon the information receivedfrom the "FROM" and "TO" strips. The desired angle of rotation θ mayalso be calculated by the microprocessor in accordance with thefollowing formula:

    θ=Tan.sup.-1 δ/λ

The microprocessor thus uses the information received from the "FROM"and "TO" strips to electronically rotate the image clockwise by an angleθ from the image's present angular orientation to the image's desiredangular orientation. Pages that are subsequently printed by the copierwill provide images that are rotated by the indicated amount.

Embodiments of the present invention enable a conventional copier tofunction as a "receiving end" facsimile printer. In one embodiment afacsimile module may be plugged into the tablet to enable fax datatransmitted over standard telephone lines to be intercepted,interpreted, and converted into displayable form. Once present on animage forming screen, the graphical data may be reproduced by a copier.The facsimile information may be directly viewed off of the imageforming screen, and thus need not be printed unless desired.

In this manner the present invention provides for both "optical fax" andconventional "hard copy fax", and permits the user to inspect atransmitted document in its entirety before deciding whether all or onlyportions are to be printed. Variations on this theme include embodimentspossessing document scanning capabilities, so that data transmission ispossible, in addition to data reception. Other embodiments may beequipped with storage means so that incoming data may be saved, thenprinted in a time-shifted fashion at another moment. The informationreceived via fax transmission may include both visual information aswell as audio information.

Examples of systems that use the present invention as a fax device arediscussed in greater detail below in connection with FIGS. 39 through52.

FIG. 39 illustrates an example of how a fax document might appeardisplayed on the image display screen 6A of a fax tablet 1000. As shownin FIG. 39, the fax document includes the image of a shirt, a body ofhandwritten text, and a telephone number. Also represented in the figureis a light gasket 2, located along the perimeter of the display element6A. The light gasket performs the function of keeping stray, ambientlight from leaking in between the display 6A of the tablet and thecopier window of the copier while the tablet 1000 is being used with acopier to create hard copy.

FIG. 40 illustrates an example of the light gasket 2 of the fax tablet1000. A portion of the tablet 1000 is shown in an enlarged view tobetter illustrate the light gasket 2. As shown, the light gasket 2 is inthe form of a soft, deformable material layer situated along theperimeter of the display element 6A, that is minutely elevated above thesurface of the display screen. The light gasket 2 may be constructed ofany suitable, flexible material (e.g. rubber).

FIGS. 41(a) and (b) illustrate an example of a procedure for retrieving"Voice Notes" from fax documents containing audio components. "VoiceNotes" and "Audio Overlays" are novel features of the present fax unit.They permit an audio component, such as a note or memo dictated in thevoice of a person sending a fax document, to be incorporated intootherwise conventional fax transmissions. The present invention enablesaudio overlays to be incorporated into hard copy fax documents. Thispermits voice notes to be accessible not only from just-received faxdocuments, but from fax documents that may have been received long ago,as well as from ones that may have been received by fax units other thanthe one currently in use. As will be seen, a preferred method employedby the present invention to incorporate audio signals into hard copy faxdocuments preserves the "look and feel" of the conventional, plain paperfax documents.

Looking now at FIG. 41(a), the fax document first appearing in FIG. 39may again be seen on the display screen 6A of the fax tablet 1000.Depicted are a picture of a shirt, a body of handwritten text, and atelephone number. Both the image of the shirt 1110a and the telephonenumber 1110b are graphically enclosed by boxes that are darkened orotherwise highlighted. These highlighted boxes indicate that there areaudio messages associated with the highlighted portions of the faxletter.

The image-forming screen 6A of the fax tablet 1000 is an ATM-styletouch-sensitive screen. By touching any of the highlighted regionsappearing on the screen 6A, the user may play back the audio messageassociated with the highlighted portion of the document. A loudspeaker(not shown) included in the tablet may be used to play back the audiomessages.

FIGS. 41(a) and (b) illustrate an example of a procedure for playingback the audio component of a fax message. By touching the highlight box1110a enclosing the image of the shirt as shown in FIG. 41(a), the usercauses the tablet 1000 to play back an audio message stating that "Thisstyle comes in sizes small, medium and large. It does not come in extralarge." By touching the highlight box enclosing the telephone number1110b as shown in FIG. 41(b), the user will cause the tablet 1000 toplay back a received audio message stating that "My fax tablet number854-5678, in case you want to transmit voice/fax mail to me."

FIGS. 42(a)-(d) illustrate an example of the document scanner facilityof the fax tablet 1000. The document scanner facility includes both anaudio sub-assembly and a conventional graphic sub-assembly. The audioassembly includes two sub-components: an audio read/write head, and an"audio patch" applicator.

In FIG. 42(a), a document input tray 1120 is shown. The user would placea document into the document input tray 1120 to have the documentprocessed by the document scanner facilities. A document input throughthe document scanner tray 1120 is transported through the scannerassembly and exits at exit slit 1121. FIG. 42(b) illustrates the removalof a part 1123 of the tablet 1000 so as to reveal the components of thescanner assembly. FIG. 42(c) shows the audio components of the scannerassembly. Audio read/write means 1126 are pictured as well as an "audiopatch" applicator 1124. The audio read/write means 1126 performs thefunction of reading from and/or writing to an audio patch as a documentis being transported through the scanning assembly. The audio read/writemeans 1126 may comprise a magnetic read/write head, similar to thoseused in tape recording equipment or to those employed to store and toretrieve data onto an from computer diskettes. The "audio patch"applicator 1124 is in essence a label applicator that applies adhesivelabels, having, for instance, a magnetic recording surface layer coatedonto the non-adhesive face, onto documents passing through the scannerpathway. FIG. 42(d) shows an optical scanner 1127 on the cut-awayportion 1123 of the fax tablet 1000. The optical scanner 1127 maycomprise a linear photodetector array similar to those found in opticalscanners currently available.

The audio patch retains information for the audio component of a faxdocument, when the document possesses such a component. In a preferredembodiment, an audio path may comprise a small strip of flexiblemagnetic material, although other types of recording media may be used(e.g. optical). The substrate of an audio patch might be constructed ofa flexible polymer, of treated paper, or it might be constructed of aflexible metal such as aluminum, onto which a recording medium may beapplied. The audio patch applicator 1124 applies the audio patch to theback of a document sheet, generated by a copier. The audio patchtechnique of the present invention enables a fax document to retain the"look and feel" of a conventional, plain paper fax document. Yet itpermits the document to retain audio information in addition to theconventional, graphical information.

An entire fax transmission, including both audio and visual information,may be received by the tablet 1000 from a remote audio-capable fax unitand stored in the tablet's memory. The visual information may be playedback on the tablet's image display screen, while the audio informationmay be played back on the tablet's loudspeaker.

FIGS. 43 and 44 illustrate an example of the process of producing hardcopy of an entire audio-fax document that includes the associated audiomessages. The tablet 1000 may first be placed on a copier 4 and plainpaper hard copy P may be generated by the copier in a manner previouslydescribed. The plain paper hard copy P (which is similar in appearanceand mechanical properties to that produced by a conventional plain paperfax machine) may then be taken from the copier 4 and fed through thescanning assembly of the fax tablet 1000. As the sheet P is being fedthrough the scanning assembly, the audio patch applicator 1124 willgenerate an appropriate audio patch AP (based on the audio informationin the tablet's memory) and will apply the audio patch AP to the back ofthe fax document P. The audio patch AP enables the user to keep therecording of the audio portion of the fax transmission.

FIGS. 44(a) and (b) respectively illustrate an example of the front andback of a page P printed by the copier 4. The back of the page P isblank and has not yet received an audio patch. The printed page P is fedinto the scanning assembly as if the user were scanning the page fortransmission. The audio patch AP is then applied to the back of theprinted page P, as shown in FIG. 44(e).

In an alternative embodiment, the tablet may simply create an audiopatch but not apply the audio patch to the back of the printed page. Insuch an embodiment, the audio patch may be manually applied to the backof the printed page as a separate procedure. The back of the printedpage may be provided with printed guide markers to assist in positioningthe audio patch.

FIG. 45 illustrates an example of a system for playing back a printedpage P having an audio patch AP. FIG. 45(a) illustrates the front of theprinted page P containing both text and graphic information. FIG. 45(b)illustrates the back of the printed page P to which an audio patch APhas been applied. The printed page P may be fed through the scanningassembly of the tablet 1000, as shown in FIG. 45(c). The text andgraphic information on the front of the printed page P may be scanned byan optical scanner 1127, in a manner well known in the art. The text andgraphic information scanned by the optical scanner 1127 may be stored inthe memory and displayed on the display image display screen 6 (see FIG.45(d)).

The audio patch AP on the back of the printed page P may be read by anaudio read/write head 1126. The audio information read by the audioread/write head 1126 may be stored in the tablet's memory and playedback through the tablet's loudspeaker.

The text and graphic information displayed by the image display screenmay include highlight boxes (see FIGS. 45(e) and (f)). Touching ahighlight box on the tablet's touch sensitive image display screencauses an associated audio message to be played back through thetablet's loudspeaker. This system enables the user to access the audioinformation of a fax transmission that may have been received yearsearlier, or that may have been received by another fax machine.

FIGS. 46 and 47 illustrate an example of how the audio portion of a faxdocument may be recorded. In FIGS. 46(a) a user's finger is shown on theupper left corner of what will be an audio highlight box. In FIG. 46(b)the user's finger is shown dragging the bottom right corner of theforming highlight box to enclose the image of the shirt. In FIGS. 46(c)the image of the shirt has been completely enclosed within the bordersof a highlight box. FIG. 46(d) portrays a completed highlight boxdefining a region of a fax document for which an audio message may nowbe recorded.

FIG. 47(b) shows an enlargement of several on-screen buttons, includinga screen button having the icon of a tape cassette. The screen buttonmay blink to indicate that a region of the document has been selected(highlighted) and is awaiting the recording of an audio message. Theuser touches the screen button having the icon of the tape cassette, toinitiate the audio recording sequence, and begins speaking into amicrophone supplied with the tablet. When the dictation of the audiomessage has been completed, the same screen button is again pressed. Anaudio overlay message has now been created for that particular,highlighted portion of the document. In FIG. 47(a) the highlight box isdarkened to indicate that an audio message is present. In contrast, inFIG. 46(d) the highlight box is clear, and is formed of dotted lines.This indicates that a region has been defined for incorporation of anaudio overlay, but that no audio message has yet been recorded.

While the audio overlay examples described above have involved humanvoice dictations, it is possible for audio overlays to comprise otherforms of acoustic material. For example, a fax document containing alist of musical recordings, such as a listing of compact discs, may havea few seconds of sample music included with each title listed. Othervarieties of acoustical information may also be with the audio overlaysof the present fax unit.

A fax tablet in accordance with the present invention may communicatevia either a cellular link or a conventional wire link. A fax tablet inaccordance with the present invention may communicate with other,conventional fax machines that do not have audio message capabilitiesand that do not have image display screen capabilities.

FIG. 48 illustrates an example of a communications protocol for faxtransmission. The illustrated protocol enables a receiving fax unithaving audio capability in accordance with the present invention to beused with both prior art transmitting fax units as well as transmittingfax units having audio capability.

As illustrated in FIG. 48, when the transmitting fax unit has completedtransmitting conventional (i.e., non-audio) fax information, thetransmitting fax unit transmits a disconnect signal ("DCN") to thereceiving fax unit. A conventional receiving fax unit will hang up uponreceipt of such a disconnect signal.

In the illustrated example, a receiving fax unit having audio capabilityin accordance with the present invention will not hang up immediatelyupon receipt of the disconnect signal. Instead, the receiving fax unitwaits for a predetermined period of time (for example, fifteen seconds).If the transmitting fax unit is an audio capable unit, then thetransmitting fax unit transmits an audio capable signal ("VOC") shortlyafter transmitting the conventional disconnect signal ("DCN"). If theaudio capable receiving fax unit receives the audio capable signal fromthe transmitting fax unit, then the receiving fax unit sends an audioacknowledge signal ("VOA") back to the transmitting fax unit. Then, thetransmitting fax unit begins transmitting the audio component of the faxtransmission. After transmission of the audio component is completed,the transmitting unit sends another disconnect signal. The disconnectsignal may be identical to the previously sent conventional disconnectsignal ("DCN"), with the receiving fax unit being designed to hang upafter receiving two cycles of disconnect signals.

When a fax tablet displays a fax transmission, it may also be programmedto label each separate highlight box with a corresponding identificationletter or numeral. FIGS. 49(a) and (b) illustrate examples of highlightboxes that are labeled with small superscript identification numerals.In the illustrated example, the highlight box containing the image ofthe shirt has been labeled with a superscript identification numeral"1", and the highlight box containing the telephone number has beenlabeled with superscript identification numeral "2". The primary purposefor such identification numerals is to allow a user to identify and playback specific audio messages without requiring the use of a touchsensitive image display screen.

FIG. 50 illustrates an example of a voice capable desktop fax unitcontaining an image display screen that is not touch sensitive. In theillustrated example, if the user wanted to hear the audio messageidentified by the numeral "2", then the user would first place the faxdocument having an audio patch in the scanning tray of the desktop faxunit. The user would then push a button on the operator's console thatsays "play voice note #02". The document would be transported throughthe scanning assembly and an audio read/write means in the desktop faxunit would read and play back the audio message identified by thenumeral "2".

FIG. 51 illustrates an example of a cradle C into which a fax tablet1000 may be placed. The cradle C may provide many or all of theelectrical connections between the tablet 1000 and the outsideenvironment. For example, the cradle C may include a phone connector forconnecting the fax tablet 1000 to a phone line. The cradle C may includea computer connector for connecting the tablet 1000 with a computer orcomputer printer. The cradle C may also include a power supply forpowering and/or charging the tablet 1000.

FIG. 52 illustrates an example of a cradle C equipped with a videocamera VC which enables the fax tablet 1000 docked in the cradle tosend, receive and display video images.

FIG. 53 shows a page D_(p) which may either be a page of an electronicdocument as it appears on the display screen of a device such as 1000,or an actual printed page. Portions of the page, as indicated by thedark bounding boxes, contain information that is electronically linkedto electronic reference information.

A characteristic of every link region, such as LR1 and LR2 in FIG. 53,is a set of information that defines its position within the document.There are many ways to represent the location of a link region. Onemethod is to employ coordinates. The zero-point of the coordinate systemmay be a reference position such as the top or left margin of thedocument page. In FIG. 53 the link region LR1 is defined by thecoordinates Xa(1),Ya(1); Xb(1),Ya(1); Xb(1),Yb(1); Xa(1),Yb(1)!,starting at the top left vertex of the bounding box and going around thequadrangle in a clockwise direction.

Position information defining a link region is obtained whenever, forexample, an audio highlight box is defined as illustrated in FIGS. 46(a)through (d). There are many ways to convey the position of a link regionbesides through coordinates, for instance with markers, embedded codes,recorded patterns, etc..

FIG. 54 illustrates how link information stored on a printed document inthe manner illustrated in FIG. 50 may be used to regenerate theelectronic bounding boxes 1110a and 1110b of FIGS. 41(a) and 41(b). InFIG. 54, D_(prn) represents a printed document. Like in FIG. 50,portions of the printed page contain information that is electronicallylinked to electronic reference data. Those portions of the page may beenclosed by printed markings having a pre-defined appearance. In bothFIGS. 50 and 54 small superscript identification numerals are used asthe delimiters of link regions.

For example, the superscript numeral 1 is used to identify the firstlink region 5011 on hardcopy page D_(prn), and the superscript numeral 2is used to identify the second link region 5022 on D_(prn). When theprinted page D_(prn) is scanned by a tablet 1000 as in FIG. 45(c), orsimilarly when it is scanned by a desktop fax unit like the one picturedin FIG. 50, electronic information representative of the image appearingon the printed page is obtained.

In FIG. 54 5025 represents the electronic information representative ofonly the bottom portion of page D_(prn). The concepts surroundingdigital image representation are appreciable to individuals skilled inthe art and the electronic representations may, for example, be in theform of bitmaps. Optical character recognition, or more generally,pattern matching, may be performed on the electronic image data. This isillustrated using image information 5025. When the image data 5025 isscanned 5027 for the presence of patterns used to denote link regionboundaries, the positions of the patterns that are successfullyrecognized 5029 are recorded 5031. The small superscript numeral 2 isidentified 5029 as a link region delimiter, and its position is recorded5031, as symbolized by the placement of the crosshair. The othercrosshairs shown in FIG. 54 correspond to, and are similarly derivedfrom, the other numeral 2 superscripts in document D_(prn).

The actual geometric shapes employed to designate link region boundariesmay be arrived at by agreement. For example, whereas in FIGS. 49, 50,and 54 the shapes used are in the form of small superscript numerals,they are by no means limited to these. Indeed FIG. 55 illustrates analternative in which the delimiters are in the form of brackets drawn byhand.

After all of the link region delimiters of a given group, for instancethose involving the superscript numeral 2, have been identified, errordetection procedures may be performed. A an example if, by convention,link regions are always confined within geometric boxes, then a softwareroutine may be included to confirm that the shape traced out by the sumof the identified link region delimiters is in fact quadrangular.

Once a link region has been identified it may be electronicallyassociated with a piece of electronic reference information. Differentlinking conventions are possible. An example of an implicit method mightbe to associate the first link region identified on a page, where"first" may be interpreted in the top-down/left-right sense, with thefirst piece of electronic reference information (electronically)associated with the document.

In FIG. 54 the link region 5010 is electronically linked to a firstpiece of electronic reference information, and the link region 5020 iselectronically linked to a second piece of electronic referenceinformation. An example of an explicit link mechanism might be toconnect the link region having the numeral 2 for delimiters on D_(prn)with the second piece of available electronic reference information,regardless of the physical position of this link area on the page.Therefore, link region 5020 could be electronically linked to the secondpiece of electronic reference information even if it were located on topof link region 5010 in D_(scr) in FIG. 54.

FIG. 55 illustrates how printed documents do not have to be originallycreated with electronically linked electronic references in mind inorder to accommodate this feature. Printed document D_(prn2) may havebeen taken from a magazine, or in the extreme may even be a historicaldocument--and can still be used with electronically linked electronicreference information.

In FIG. 55, 5041 denotes a designated first link region, and 5042denotes a designated second link region. The link regions are identifiedby brackets 5050, which may be hand drawn 5055. It is not a requirementthat the link area identifiers be brackets. For instance, thesuperscript numerals of FIG. 50 and FIG. 54 may also be penciled in byhand. Similarly, the identifiers do not have to be manually drawn. Theymay be stamped onto D_(prn2), or superimposed using overlays or othertechniques, etc.

The concept illustrated in FIG. 55 is the addition of link regionidentifiers in a step separate from the generation of the printed matterto which the identifiers are being added. After D_(prn2) is scanned asin FIG. 45(c), pattern matching may be performed as described inconnection with FIG. 54. Bounding boxes such as 5010 and 5020 may begenerated, and these may be electronically linked to electronicreference information in the manner practiced throughout thisdisclosure.

FIGS. 56(a) and (b) illustrate the storage of a remote link toelectronically linked electronic reference information in an electronicdata storage medium capable of being included on a printed documentD_(pn). The term "Electronic Data Storage Medium" (or EDSM) offers amore accurate description of the device element that is capable of beingincluded on printed documents to retain or to convey electronicallylinked electronic reference information. Where the electronic referenceinformation is comprised of electronic audio information, the EDSM mayalso be referred to as an "Audio Patch". The electronically linked audioinformation of FIG. 41 was stored in the audio patch AP of FIG. 44(e)with a local link.

It is not a limitation, though, of the links stored in the ElectronicData Storage Media described herein to be of the local variety. As shownin FIG. 43(a) the apparatus 1000 can communicate with a variety ofdevices situated at remote locations via wired (line) or cellularconnections. In response to such capability, it will sometimes benecessary to store remote links in the EDSM so that (portions of)printed documents may reference information that may not be saved on theEDSM included with the document.

FIG. 56(a) shows an electronic data storage medium AP capable of beingincluded on a printed page D_(pn). The (EDSM) electronic data storagemedium includes link information (LI) and electronic referenceinformation (ERI). A local link 5060 is shown which points to referenceinformation 5061. In FIG. 56(a) both the link information and theelectronic reference information are depicted as being stored in aseries of fields. It is preferable that at least the link information besaved in this manner, or in some other comparable fashion whoseobjective is to enable each piece of electronic reference information tobe accessible independently of any extraneous data.

As shown in FIG. 50 the storage structure should permit convenientaccess to the electronic reference information even in the absence of a(Graphical User Interface) GUI Pointer (e.g.. touch sensitive screen,mouse, electronic pen, etc.). Furthermore, it should be possible toaccess any piece of electronic reference information without the need torefer to the original body of electronic data having the electroniclinks to the electronic reference information. The structure of FIG.56(a) permits this.

Two remote links are shown in FIG. 56(a) and they are indicated as 5064.The first remote link is depicted as pointing to electronic referenceinformation residing in a mass storage device. Mass storage device 5071may be a hard drive included with the desktop fax machine of FIG. 50.Or, it may be a PCMCIA card present in the tablet 1000 of FIG. 45. Theseare but two examples.

The remote link information recorded in AP includes node accessrequirement information (NAR). The node access requirement informationmay include instructions, passwords, etc. in addition to the identitiesof any hardware or software which may be needed to enable the electronicreference information saved in the mass storage device 5071 to beretrieved.

The saved link information also includes the remote address (R₋₋Address) of the electronic reference information such as the physicaland logical drive names, directory paths, etc. where they are located.Information identifying the portion of the document making the referencecall may also be supplied. This is represented as 5069. A second remotelink is shown in FIG. 56(a). This time it points to electronic referenceinformation located on a network or on a remote host 5073.

The tablet 1000 may be in communications with such devices as seen inFIG. 43(a). To illustrate this, the video sequence in FIG. 52 mayconstitute information deposited in a voice mail box which iselectronically linked to, for instance, the name appearing on a printedcopy of the little girl's report card. The tablet 1000, docked in videocradle C (as in FIG. 52), may access the voice mail box (as in FIG.43(a)) using a remote link stored in the electronic data storage mediumAP.

As another example, in FIG. 56(b), the execution of the command "PlayNote #2", 5076, causes the second field of link information stored in APto be read. Here, the information comprises a remote link to an on-lineservice 5073. NAR2 of this stored link may include the phone number ofthe desired on-line service, as well as any communications protocols orpasswords which may be needed to establish the connection.

"R₋₋ Address 2" may include the sequence of commands required to enterthe portion of the on-line service providing the desired referenceinformation. Or, in an abbreviated arrangement which requires lessstorage space on the electronic data storage medium AP, the NAR framemay indicate the name of a complete applications module or batch filewhich must be available to the document playing apparatus, in order forthe sought-after remote-linked reference information to be obtained.

5077 in FIG. 56(b) represents a message which may be issued by anapparatus that is playing a document D_(pn) during the time that aremote connection is being established via a remote link. Also, 5076 maybe an actual voice command spoken by the person playing document D_(pn)to, for example, a tablet 1000. This may be the case where the tablet1000 possesses voice recognition capability.

FIG. 57 shows the use of a Non-Standard Facilities frame by a facsimileapparatus, to tell a comparable apparatus how it will be conducting amodified facsimile session. The NSF frame is included as an optionalfeature of the CCITT fax communications protocol. In FIG. 57, the NSFdata may include codes for designating the transmission mode, link mode,special characters in use, etc.. For example, the document image data aswell as the linked reference data may be encoded as fax data andtransmitted. Whereas the document image data dispatched in this fashionmay still be intelligible to an unsuspecting receiving apparatus, thelinked reference data may be useless unless the receiving apparatus isaware that such a (modified) transmissions scheme is being employed.

FIGS. 58(a-d) show the communications protocol of FIG. 48 in greaterdetail.

FIG. 58(a) outlines the general scheme of the communications protocol.The method utilizes two communications sessions, executed typically,although with some exceptions, in back-to-back sequence. The firstsession is referred to as the Primary Communications Session, and thesecond session is referred to as the Secondary Communications Session.

During the Primary Communications Session the document will typically betransmitted, by way of a conventional data transmission technique suchas one of the CCITT fax protocols. During the Secondary CommunicationsSession the electronically linked electronic reference information willtypically be transmitted. This was seen in connection with FIG. 48 whereelectronically linked audio information was sent. While the PrimaryCommunications Session may involve one of the CCITT fax protocols, thisis not a limitation of the present method, whose objective is to enablebi-directional communications of electronically linked electronicreference information with existing communications schemes not intendedto accommodate this capability.

The document which is transmitted during the Primary CommunicationsSession of FIG. 58(a) may optionally include link information embeddedwith the document information. The electronically linked electronicreference information which is transmitted during the SecondaryCommunications Session will usually include link information; althoughthe link information need not be conveyed in an explicit form. That is,the link information may be manifest in the transmission and storagestructure of the electronically linked electronic reference information.

For example, a pre-defined period of silence may separate two successivepieces of electronic reference information during their transmission;and, when stored in the electronic data storage medium which will beincluded on a printed document, each piece of electronic referenceinformation may be assigned a separate storage field. For instance, thethird piece of electronic reference information may be located in thethird storage location.

FIGS. 58(b)-(d) illustrate how the presently taught communicationsprocedure enables otherwise conventional fax transmissions to includeelectronically linked audio information. In FIG. 58(b) a documentcontaining electronically linked electronic audio information is beingsent using an existing fax protocol, such as the CCITT Group 2 or Group3 facsimile communications protocol. In the primary communicationssession a facsimile image of the document is transmitted in a mannerconsistent with the CCITT standard. Electronic data representative ofthe original document image is transmitted.

A novel (optional) addition of the present method involves transmittingelectronic data not just representative of the original document, butrepresentative of the original document with superimposed link regiondelimiters like the ones shown in FIGS. 49, 50, 54 and 55. If thetransmitting station comprises an apparatus such as the communicationstablet 1000 in FIG. 43, or the desktop fax unit shown in FIG. 50, thenthe link information may be electronically superimposed onto theelectronic representation of the original document by the apparatus. Inthe case where the transmitting station comprises a conventional,current-art fax machine, the above-mentioned link information may behand drawn as described in connection with FIG. 55.

The linked audio information communicated during the secondarycommunications session in FIG. 58(b) is being transmitted in what willbe referred to as Voice Mode. After the fax transmission of the primarycommunications session has been successfully completed, the sendingfacsimile apparatus disengages the line. If the sending stationcomprises a conventional fax machine, then it simply hangs up. If thesending station is an apparatus such as the tablet 1000 of FIG. 43(a),then the apparatus 1000 does not hang up but instead disengages the useof the fax component of the equipment. The linked audio information isthen sent as "voice data". If the sending station employed aconventional fax machine to transmit the fax portion in the primarycommunications session, then a telephone on the same line as that faxmachine may be used to send the linked audio information.

To illustrate the method, consider sending the two linked audiosequences depicted in FIG. 47. After the fax machine has hung up, thesender may press a key on the key pad of the telephone she is using--forinstance, the "*" key. In accordance with the DTMF (Dual ToneMultifrequency) method, this generates a tone which is actuallycomprised of two frequency components: 941 Hz and 1209 Hz. A receivingapparatus such as 1000 of FIG. 43(a) will detect the tone and interpretit as designating the start of transmission (in the voice band) ofelectronically linkable electronic (audio) reference information.

The remote apparatus may respond with a voice prompt such as "Ready toreceive". The sender may then press the "#" button followed by thenumber of the linked electronic reference that will be sent, for example"1", and again followed by the "#" button. The sender then dictates thevoice note over the phone, for example: "This style comes in size small,medium and large. It does not come in X-large." After the dictation ofthe linked audio message has been completed the "#" button is againpressed. The first linked voice note has now been communicated to theremote, receiving station.

Similarly, to send the second linked audio reference shown in FIG. 47,the sender would press "#2#" and speak into the phone: "My fax-tabletnumber is 854-5678, in case you want to transmit vice/fax mail to me."This is again followed by pressing the "#" button on the telephonekeypad. After all of the linked electronic audio references have beendictated, the sender presses the "*" button on the telephone keypad. Thereceiving apparatus 1000 will respond by offering a set of options:"Would you like to replay your voice notes?", etc.. Some versions of thereceiving apparatus may hang up upon receipt of the terminal "*" tone.

The audio passages received by the apparatus 1000 may be stored in amemory unit included with the apparatus, whether the memory is in theform of semiconductor RAM, magnetic or optical disk drive, or magneticor optical tape, etc.. The apparatus may also proceed to immediatelystore the audio information in an electronic data storage medium APwhich can be included on a printed document, as illustrated in FIG.44(e).

In the manner just described, it is possible to create and to send,using a conventional facsimile machine, a document having electronicallylinked electronic reference information (e.g. audio passages), to areceiving apparatus such as the tablet 1000 of FIG. 41 or FIG. 43(a).

The present method also accommodates the situation where the sendingstation comprises an apparatus such as the tablet 1000 of FIG. 41, andthe receiving apparatus comprises a conventional fax machine withtelephone answering machine capability. In such a case the tablet 1000will first transmit the document image information to the receivingstation as an ordinary facsimile transmission. The actual electronicrepresentation of the document that is transmitted may include linkregion delimiters such as those depicted in FIG. 49 superimposed ontothe original document. In this way embedded links may be stored on theprinted document such that portions of the printed document may beelectronically re-linkable to electronic reference information at alater time. (In some versions machine readable optical codes may besuperimposed onto the original document.) This permits the linking stepto be time-shifted.

After the document has been successfully transmitted to the receivingstation, both stations hang up. The sending station then redials thereceiving station and waits for the outgoing announcement played by thetelephone answering machine component of the receiving station to becompleted. When the sending apparatus detects that the audio signallevel coming over the telephone line from the receiving apparatus hasdropped, the outgoing announcement is interpreted as complete. Thesending apparatus then begins playing to the answering machine of thereceiving station(over the telephone line) the audio passages associatedwith portions of the already faxed document . The "#" tone may be playedover the telephone line by the sending apparatus between successiveaudio passages.

Some versions of the apparatus may identify each audio passage byincluding the audio note number, for example, by playing the "#2#" toneprior to playing audio note 2. Other versions of the apparatus may evenidentify each audio note by playing an identifier using a syntheticvoice, for example, "Voice Note 3 follows", followed by playing voicenote 3.

The present method therefore enables an ordinary fax machine working inconjunction with an ordinary telephone answering machine to send,receive, store, and to generate printed documents having portions thatare electronically linked to electronic reference information.Furthermore, by re-inputting the information received by the telephoneanswering machine into the apparatus 1000 of FIG. 41, and by scanning(as in FIG. 45(c)) the fax document received by the ordinary fax machineinto the apparatus 1000, the linked audio information may again beaccessed as illustrated in FIGS. 41(a) and 41(b).

FIG. 58(c) illustrates a variation of the communications method in whicha document having electronically linked electronic reference informationis transmitted using three communications modes. First, the documentitself is transmitted using, for example, the CCITT Group 3 faxprotocol. The transmitted document may include embedded link informationin any of the ways described earlier. Next, the electronically linkedelectronic reference information (for example voice notes) istransmitted using two modes. A data mode is used to send data includinglink data and a voice mode is used to send audio passages.

In accordance with FIG. 58(c), two pieces of equipment (like 1000 ofFIG. 43(a)) may implement a multimode transmissions scheme tocommunicate documents having electronically linked electronic referenceinformation. After the document portion has been successfullytransmitted using, for example the CCITT Group 3 fax protocol, a datamodem using predefined settings may be used to transmit data such asthat shown in FIG. 56(a) stored in AP. Other forms of data may also besent, such as computer files.

Following the data mode a transition to voice mode may be made, in whichthe data modem releases the line without hanging up. In the AT CommandSet a modem command such as "AT+FVo", "AT+FVoice", and "ATG" will bringabout a transition to voice. Spoken information may be transmitted usingvoice mode, as described above in connection with FIG. 58(b). The "VOC"command of FIG. 48 may be transmitted at the start of data mode in FIG.58(c) by the data modem of the sending station.

One form the "VOC" command may take is a set of "*" tones of the form"*N", where "N" is a one digit number from 0 to 9. For example "*1" maycomprise a tone of 941 Hz and 1209 Hz followed by a tone of 697 Hz and1209 Hz. Then minimum duration of each tone may be 75 milliseconds, andthe minimum interdigit interval may also be 75 milliseconds. The tone of"*" without a following numeral may default to signify that a transitionto voice mode will be made. This was seen in connection with FIG. 58(b)where, after the fax hardware released the telephone line, the "*" tonewas used to signal the start of dictated voice notes.

The tones associated with "*3" (941 Hz and 1209 Hz plus 697 Hz and 1477Hz) indicate to the tablet 1000 of FIG. 43 that data mode is being used.This simply means that a data modem will be transmitting data over theline, which is still active, although the fax component has alreadydisconnected from the line.

FIG. 58(d) shows a version of the communications scheme in which twomodes are employed: a facsimile mode and a data mode. The documenthaving the electronic links to electronic reference information is sentusing the facsimile mode; and the linked audio information is sent usingthe data mode. The fax portion of the scheme is as described earlier.The data portion is similar to the data portion in FIG. 58(c), exceptthat in FIG. 58(d) the audio passages themselves are transmitted aselectronic data. This requires that the audio information, which mayhave been obtained as illustrated in FIG. 47(b), be in digital form. Theconversion of audio signals into digital form is well known in the artand, other than placing greater requirements on the data processing andtransmissions overhead of the present method, offers many conveniencesto the user. It is possible to transmit many forms of electronic databesides electronic audio or video information. For example, computerfiles may be sent instead.

Using the methods described above it is possible to transmit documentshaving electronic links to a vast array of types of electronic referenceinformation, and to do so while making use of existing equipment whichotherwise would not have this capability. Variations of the methodsdescribed above are also possible. For example, the electronic referenceinformation may be transmitted prior to the transmission of thefacsimile image.

Finally with regard to the presently taught communications method, it ispossible for the Non-Standard Facilities (NSF) feature which is a partof the CCITT facsimile protocol to be utilized to enable an apparatussuch as the tablet 1000 of FIG. 43 to communicate with another similarapparatus. It is also possible for the audio information, whichcomprises link information and audio data, to be transmitted as NSFdata.

For example, a non-standard setup may involve sending audio data encodedas facsimile data. In some instances this method may even be desirable.However, to enable bi-directional compatibility between the devicespresently taught and conventional devices such as ordinary fax machines,the scheme described in conjunction with FIGS. 48 and 58 is believed tobe better. The NSF feature which is an integral part of the CCITTfacsimile standard may be resorted to when it is known that comparableequipment will be used at both sending and receiving stations. The NSFfeature may also be applied to supplement the methods described herein,as between two similar devices such as 1000 in FIG. 43(a).

FIG. 59 illustrates the synchronization of information storage (into anEDSM to be included on a printed document) with the formatting of theassociated document. 5100 represents electronic data having portionsthat are electronically linked to electronic reference information. Someof the electronic data, represented by 5110, will be used to generateprinted copy. Other parts of the electronic data, indicated by 5120,comprise electronic reference information and link information. Pathway5113 illustrates what may occur without synchronization. AP is anelectronic data storage medium capable of being included on a printeddocument. The data printed onto page Pg includes three reference callsto electronically linked reference information: 5134, 5136, and 5138.5124 represents the electronic reference information associated withreference call 5134; 5126 represents the electronic referenceinformation associated with reference call 5136; and 5128 represents theelectronic reference information associated with reference call 5138.After storing the data associated with 5124 and 5126 in the electronicdata storage medium AP, the remaining storage space 5127 on AP isinsufficient for retaining the data associated with electronic referenceinformation 5128.

Pathway 5117 illustrates formatting the hardcopy and partitioning thestorage of electronic reference information in synchronization. Printedpage Pgx has been formatted to include the information containingreference calls 5134 and 5136. The electronic reference information 5124and 5126, which are respectively linked to 5134 and 5136, is stored on afirst electronic data storage medium AP capable of being included onpage Pgx. The data which includes the reference call 5138 is printed onanother page Pgy. The electronic reference information 5128 which iselectronically linked to 5138 is stored in a second electronic datastorage medium capable of being included on the second printed page Pgy.

FIGS. 60(a) and (b) illustrates an example of how information may beorganized so that it will not be necessary to acquire an entire body ofelectronic data in order to access to the electronic referenceinformation that is electronically linked to portions of that body ofelectronic data.

FIG. 60(a) represents the electronic data having portions that areelectronically linked to electronic reference information. Three linkregions LR_(a), LR_(b), and LR_(c) are shown within a "primary" block ofelectronic reference information ERI_(p). Link region LR_(a) is linkedto electronic reference information ERI_(a)(1). Similarly link regionLR_(b) is linked to ERI_(b)(1), and LR_(c) is linked to ERI_(c)(1).ERI_(b)(1) itself includes a link to an electronic reference ERI_(b)(2).

Given the electronic data shown in FIG. 60(a), the information shown inFIG. 60(b) may be (electronically identified and) obtained. The table ofFIG. 60(b) is offered only as an example. Other organizational schemesare possible.

The first horizontal row in the table identifies the access requirementsof each of the nodes in the complete body of electronic data shown inFIG. 60(a). The first row may be considered a Node Access RequirementHeader. It may be more efficient for a printed document that includes adata storage medium containing complex, linked electronic referenceinformation to have the information shown in FIG. 60(b) readilyaccessible. A table as shown in FIG. 60(b) may be stored in theelectronic data storage medium that is included on the printed document.It may also be retained in the memory of a device such as 1000 in FIG.41, or in the memory of the desktop fax apparatus of FIG. 50.

The Node Access Requirement Header may reveal the requirements whichmust be met in order for each of the electronic references to beaccessed; for example, any supplemental hardware or software needed bythe electronic reference data. Each column in the table furnishesinformation used to access (and to establish the link to) a piece ofelectronic reference information. For instance, looking at the secondcolumn, the first entry identifies any special hardware or softwarewhich may be needed to access the electronic reference informationlabeled ERI_(a)(1) in FIG. 60(a); and the second entry enables a link toERI_(a)(1) to be established. Each piece of electronically linkedelectronic reference information may be retrieved independently of anyother information contained in the original body of electronic data.Similarly, GUI pointers are not required for access to be achieved. Theexample supplied in FIG. 50 may make use this feature: "Play Voice Note#02".

FIGS. 61(a) and (b) illustrate an apparatus for playing printeddocuments that have electronic data storage media included thereon, suchas the page P of FIGS. 45(a) and (b). In FIG. 61(a) a printed document Pis being inserted into the playing apparatus 5225. Scan window 5230includes an optical or magnetic read head for reading the electronicdata storage medium AP included on the rear surface P_(b) of the printeddocument P (as in FIG. 45(b)). Cover plate 5235, which is attached toplaying apparatus 5225 by a hinge 5233, comprises a liquid crystaldisplay means. Cover plate 5235 is substantially optically transparent,and thus when closed shut over a printed document P which has beenplaced into the playing apparatus 5225, will permit information formedon the printed document to be viewed.

Images generated on the liquid crystal display means 5235 may besuperimposed onto images formed on the printed document P. This allowsgraphics derived from electronic reference information linked toportions of the printed document P to be overlaid onto the printeddocument. FIG. 61(b) illustrates a printed document being opticallyscanned by playing apparatus 5225. The optical scanning mechanism 5240may include a linear photodetector array like 1127 in FIG. 42(d). FIG.61(b) also shows a backlit portion 5231 of the playing apparatus 5225.

FIG. 62 shows the playing apparatus 5225 playing a printed document P.The dark portion of the graph 5258 in the center of the page is formedon the liquid crystal display means of the cover plate, and is beingsuperimposed onto the printed information of page P. The two arrows arealso electronically formed images overlaid onto the printed page.

The data and instructions for forming these images comprise electronicreference information saved in an electronic data storage medium affixedto the page P. The video sequence 5250 pictured at the bottom of theapparatus 5225 is similarly being formed from electronic referenceinformation on the liquid crystal member. The person shown in the videosequence is explaining the printed document P. The audio portion 5255 ofthe video sequence 5250 is being played in coordination with thegraphical plot 5258 and the appearance of the arrows. The lower arrow ispointing to the part of the graph on the printed page, which represents"the department's prediction". The upper arrow is pointing to the partof the graph formed on the liquid crystal display means. It representsthe "actual outcome" spoken of by the person in the video sequence 5250.

FIG. 63(a) shows a page of a hardcopy document that will be sent by aconventional facsimile machine having only monochrome capability to thereceiving apparatus of the present invention. A conventional facsimiletransmission protocol will be used, such as the CCITT group 3 facsimiletransmission protocol. In FIG. 63(a) 8400 represents one page out of amulti-page hardcopy document which can contain any number of pages. Atthe top of the page is the word "URGENT", 8440. In this illustration thesender wishes to emphasize to the recipient of the fax transmission thatthe incoming document is of a high priority. Thus, the sender would likethe word "URGENT", 8440, to be in red in the received document.Similarly, FIGS. 63 (a) and (b) contain a block of text 8420 which theauthor of the document (sender) considers to be of particularsignificance. To signify this, 8420 is desired to be in green whenreceived. The color selections mentioned are merely for illustrativepurpose, and there is no substantive relevance to the particular colorschosen.

FIG. 63(b) shows graphical identifiers 8470 being drawn by hand onto thedocument image of the hardcopy page. In actuality, any means forgraphically adding the identifiers may be employed. For example, theymay be printed or stamped onto the document, or even added in the formof adhesively applied appliques. Also several different schemes mayconcurrently be employed to accomplish the task. When the document is inelectronic format, the graphical identifiers may be electronicallyadded. In FIG. 63(b) the graphical identifiers 8470 are shown beingmanually marked 8480 onto the document page 8400. The graphicalidentifiers are depicted as comprising delimiters in the form of doublebrackets. Double brackets are but one possibility, and not a limitationof the invention. For example, a highlight marker may also be used, suchas one which renders the highlit portion a particular color or shade ofgray. In such instances, the hardware must be able to accommodate imagescontaining those chromatic values; for example, the document scannerused to input the hardcopy document for transmission must be capable ofdetecting color or grayscale. The delimiters 8470 graphically added tothe document image designate which portion of the received document isto have an operation performed on it. A variety of different "grammars"or conventions for a document image processing language are possible,and it thus should not be construed that the ones presently suggestedare limiting.

In FIGS. 63(a) and (b) there is a portion of the document image, 8450,that will be electronically linked to the photograph 8460 of FIG. 65.Portion 8450 is located on the first page of the two page fax documentbeing sent to the receiving apparatus, and 8460 is located on the secondpage. After they have been electronically linked to each other by thereceiving apparatus it will be possible to access either linked portionwithout having to jump between the actual pages they are located onwithin the document. In lengthy multi-page fax documents this featurecan be very convenient in facilitating the review of thematicallyrelated material.

In FIG. 64, 8410 corresponds to hardcopy document page 8400 of FIG. 63after it has been completely marked up for transmission on aconventional (monochrome) facsimile machine. The sender wishes the word"URGENT", 8440, appearing in the top left corner of the document page,to be in red when received. FIG. 64 illustrates one example of a grammarfor indicating the operations to be performed on the document portions8440, 8420, and 8450. The different symbols used to designate differentoperations may be considered the grammar for a document image processinglanguage. For example, the portion of the document designated 8440 willbe translated into red as follows: first, portion 8440 (containing theword "URGENT") is placed in delimiters 8470 to signify to the receivingapparatus that the designated portion has to have an operation performedon it; and second, a transfer function of "R" (for red) is placed in thelower right corner of the delimited region. The black pixels in thedelimited portion of the document image will be translated to redpixels. Numerous transfer functions 8490 are available for bringingabout translation, or re-mapping, of document image attributes. The onejust described in connection with the translation of pixel color is butone example.

There is another portion of the document image, 8420, in FIG. 64 whichwill undergo color translation. In the case of 8420 the black pixels inthe designated document portion will be remapped to green. The imageattribute transfer function 8490 for the delimited portion containing8420 is "G". Thus when hardcopy page 8410 of FIG. 64 is received by theapparatus of the present invention, the document portion 8420 willappear in green; even though the conventional fax machine which sent thepage, sent it in black.

There is also a third document portion, 8450, shown in FIGS. 63 and 64.Unlike the just-described document portions 8420 and 8440, portion 8450will not undergo color translation. Instead, it will become the firstelement of a linked sequence. The numeral "1" in the upper left cornerof the delimited portion containing 8450 is referred to herein as aconnection identifier 8500. Such connection identifiers are used in oneembodiment of the present invention to electronically link portions ofmulti-page documents to each other. Two types of connection identifiersare utilized by the embodiment of the present invention describedherein--connection identifiers that identify the sequence to which thedocument portion belongs, and connection identifiers that identify theposition of the particular document portion within its sequence.

Linked sequences are a novel feature of the present invention. Theyoffer a convenience to recipients of multi-page fax documents when thosedocuments are received by the apparatus of the present invention. In amulti-page fax document there may be portions of one page which bearrelevance to specific portions of other pages. At present, one can onlyinsert a written note such as "(See page 4, lines 55-67)" in thevicinity of the point in the multi-page document making the reference tothe indicated portion of the document. The method now taught enables aplurality of linked sequences to be defined. Each linked sequence beingcomprised of a collection of document portions out of the completemulti-page fax transmission. Each of the individual portions within anygiven linked sequence presumably possesses some degree of thematicrelevance to one another. When any given page is displayed the linkedportions are identifiably presented in highlight boxes, within marquees,or in other manners known in the art of graphical user interfaces. Whena linked portions is selected, such as by clicking a mouse with thepointer over the linked portion, a window will open containing theportion of the multi-page fax document which constitutes the nextelement in the linked sequence. Forward and reverse controls enable auser to advance or retreat to the next or previous element in the linkedsequence. The present system thereby permits conventional facsimilemachines to send multi-page fax documents containing a plurality ofportions of special thematic relevance, said portions beingelectronically linkable to each other such that from any given linkeddocument portion, all other linked portions may be conveniently accessedfor review. In current systems, in contrast, the recipient of themultipage fax document would have to read the "See page xx, line yy"note, manually advance to the referenced page, and then inspect thejust-advanced-to page in order to manually find the position beingreferred to.

FIG. 64 illustrates one example of the use of delimiters and graphicalcodes to indicate specific operations to be performed on designatedportions of a document. The particular delimiters and symbols used arenot critical to the invention, all that is necessary is that the senderand recipient of the document agree in advance on how portions will bedesignated and the particular symbols used to identify the operation tobe performed on the portion. FIG. 65 shows the second page 8430 of atwo-page fax document. Double bracket delimiters 8470 have beengraphically added to the document image to identify a portion 8460 ofthe page that contains a photograph with a caption. The photograph andits caption will be the second element of the same linked sequence, inwhich 8450 of FIG. 64 was the first element. There is only one linkedsequence in the present example, but this is not a limitation of theinvention, which supports a plurality of linked sequences.

In FIG. 65, the portion of the document in delimiters, 8460, is to beelectronically linked to electronic reference information in the form ofan audio passage in the manner previously described in connection withgenerating electronic links to electronic reference information. A linkidentifier 8520, shown here as numeral "1", has been placed in the upperright corner of the delimited portion 8460 to signify that the delimitedportion is to be electronically linked to the acquired piece ofelectronic reference information identified (in any of the ways alreadytaught herein) as "electronic reference number 1". In the present casethe electronic reference information will be in the form of electronicaudio signals, such as a musical swatch, which will be accessible forplayback from 8600.

The left side of the delimited portion of the document image in FIG. 65contains a connection identifier in the upper left corner to identifythe sequence number 8500 of the particular linked sequence. In theexample there is only one linked sequence, though this is not alimitation of the invention. The sequence number 8500 of documentportion 8460 in FIG. 65 and the sequence number of document portion 8450in FIG. 64 are both "1". This defines both portions of the document asmembers of "Linked Sequence #1".

The numeral in the bottom left corner of the delimited portion 8460 is aconnection identifier that identifies the position of the portion 8460within the particular linked sequence and may be referred to as thesequence element I.D. number 8510 of linked portion 8460. In FIG. 65,the linked portion 8460 has a sequence element I.D. number 8510 of "2"which defines portion 8460 as the second element in the linked sequence.In FIG. 64, no sequence element I.D. number is shown for documentportion 8450 because the first element of a linked sequence encounteredin the document by the receiving apparatus is implicitly interpreted ashaving a sequence element I.D.number of "1". If portion 8450 of FIG. 64had been provided with a sequence element I.D. number by the sender ofthe document, the default value of "1" would be overridden by it.

Thus, in FIGS. 64 and 65, a document image processing language has beendefined as follows: double brackets are used to designate (delimit)portions of the document to be either transformed in some way, linked toanother portion of the same document, or linked to electronic referenceinformation. In the present example, the brackets delimit rectangularportions of the document, but this is not a limitation of the presentinvention. A symbol present in the lower right hand portion of thedelimited portion indicates an operation to be performed on the text (orfigure) within the brackets. The symbol (in the present example, anumeral), in the upper left hand corner is the connection identifierthat identifies the particular linked sequence to which the delimitedportion belongs. The symbol in the lower left hand corner (also anumeral in the present example) identifies the sequenceelement-I.D.-number, i.e., the position of the delimited portion withinthe linked sequence identified by the connection identifier in the upperleft hand corner. Finally, the symbol in the upper right hand corner(also a numeral in the present example) is a link identifier todesignate a link to electronic reference information as previouslydiscussed. This document image processing language is intended to beexemplary only--the way in which the portion of the document isdelimited, the particular delimiters used, the locations of the symbols,and the symbols themselves may be chosen according to any conventionagreed upon by the sender and recipient of the document image.

FIG. 66 shows how the marked-up document 8410 of FIG. 64 would appearwhen received by the apparatus of the present invention. Hardcopydocument 8410 was transmitted to the receiving apparatus of the presentinvention by a conventional (monochrome) facsimile machine, using astandard facsimile transmission protocol. 8530 is the electronicrepresentation of the image of hardcopy document 8410 of FIG. 64, as itmight appear on the screen of an image display device 8580. 8540 is 8440in red and 8550 is 8420 in green. 8560 is a highlight box containingdocument portion 8450.

FIG. 67 shows what happens when highlight box 8560 of FIG. 66 is"selected", such as by clicking on it with a mouse, or by touching itwhen the display device includes a touch sensitive screen--a window 8590is opened. The window is used to view the document portions that arewithin the same linked sequence as portion 8560. In the present example,when highlight box 8560 is accessed, portion 8460 of FIG. 65 appears inwindow 8590 on the image display device 8580 in electronic form 8600.But 8600 is also electronically linked to an electronic audio passage aspreviously discussed. Controls 8610 are thus provided for controllingthe aural reproduction of the linked audio information by the audiospeakers 8640.

In the example supplied herein, there is only one linked sequence, andthat single linked sequence contains only two items: document portions8450 and 8460. Still, controls 8620 and 8630 are provided for advancingto the next linked portion and retreating to the previous linkedportion. In the present case, pressing 8630 when 8600 is in window 8590will cause an electronic representation of 8450 to appear in place of8600. And, pressing 8630 again will cause the image displayed in window8590 to cycle back to 8600. Notice that in FIG. 67 the electronicrepresentation of document image 8530 is still present in the backgroundof window 8590. This is not essential and can be altered throughadjusting user preference settings.

The transformation of the text within a designated portion of a documentimage is described with reference to FIGS. 68 and 69. FIG. 68(a) shows aset of pixels that might comprise a small part of a digital image. Thenumbers beneath each pixel denote the data that represents the pixel.For purposes of illustration each pixel has been represented by athree-bit data word. It should be recognized that the present inventionis in no way limited to 3-bits/pixel. As persons skilled in the art canappreciate, an 8-bit, 24-bit or other size representation may equally beused. In actuality only a single bit is needed to represent themonochrome bi-level bitmap image of FIG. 68(a), as only two states arepresent: white (0) and black (1). As an example, pixel 8650 is shown asa white, or non-printing pixel. The data 8670 associated with pixel 8650contains all 0's because the pixel is white or non-printing. Pixels suchas 8690 and 8660 are black, or printing pixels, and are represented bythe data word "001".

FIG. 68(b) shows the pixel-wise translation of image attributes frommonochrome to color in accordance with one embodiment of the presentinvention. When the data associated with pixel 8690 of FIG. 68(a) istranslated from "001" to "010", the image attribute of pixel 8690 istranslated from black to the color (or pattern) identified as 8700 inFIG. 68(b). Likewise when the data associated with pixel 8660 of FIG.68(a) is translated from "001" to "100", the color (or pattern) of 8660is re-mapped from black to the color (or pattern) of 8770 in FIG. 68(b).Several additional examples of image attribute translation are suppliedin FIG. 68(b) but without further discussion.

FIGS. 69(a) and (b) illustrate the transformation of pixels in areceived image (FIG. 69(a)) to a displayed image (FIG. 69(b)). FIG.69(a) shows the monochrome bitmap of the word "Hi". All of the blackpixels which comprise the letters "H" and "i" are represented by thedata word "001", which is consistent with what was seen in FIG. 68(a). Amonochrome bilevel representation of a page of a conventional faxdocument containing the word "Hi" can be constituted in the mannerillustrated in FIG. 69(a). FIG. 69(b) shows what happens when all pixelsrepresented by data of a first form "001", are translated to data of asecond form. The letter "H", 8790 in FIG. 69(a), has all of its "001"pixel data words translated to "100" data words. 8800 in FIG. 69(b)shows the consequence of this translation of the associated image data.Similarly, the letter "i" of FIG. 69(a), 8810, has all of its "001"pixels translated to "110" data words. The result is a letter "i" withthe image attribute 8820 shown in FIG. 69(b). Typically, imagetransformation is not done on an individual character basis asillustrated in FIG. 69, but transformation on a character basis ispossible using the exemplary document image processing language of thepresent invention. Alternatively, rather than displaying the documentimage on a display device such as a computer monitor, it may bedisplayed in hard copy format using a color printer.

While the image data of pixels shown in connection with FIGS. 68 and 69were depicted as being comprised of 3-bit code words, this was only forpurposes of simplifying the illustration. Longer data words may be used.Indeed, longer data words are preferred. Longer code words permit awider range of possible image attributes to be supported.

FIG. 70 shows some examples of the use of a document image processinglanguage to convey instructions or operations to be performed ondesignated portions of document images. Double bracket delimiters 8470are used to signify that the portion of the document image within thedouble brackets comprises a designated portion upon which an operationis to be performed. It should be understood that this represents but onepossible way to identify designated portions--any predeterminedattribute of the portion of the document image may be used. For example,a convention may be established whereby bold-face type indicates thatthe text should be displayed in blue, italics indicate that the textshould be displayed in red, etc. In this manner, neither delimiters norcodes indicating the color for the delimited text need to be added tothe document image.

In FIG. 70 the designated portion DP1 is to have its black pixelstranslated to red as indicated by a transfer function 8490 identified as"R." Similarly, designated portion DP2 will have its black pixelstranslated to green as indicated by a transfer function 8490 identifiedas "G." Transfer function identifiers R and G convey informationpertaining to the transformation of image attributes from a first formto a second form by translating, or re-mapping, image data associatedwith the specific pixels within the designated portion of the documentimage as discussed above. For example, the transfer function identifiedas R may translate image data words from a value of "001" to a value of"111."

In connection with DP3 of FIG. 70, transfer functions 8490 are placed inposition (D), in the lower right corner of the designated portion of theimage. Connection identifiers are placed on the left side of designatedportions, in positions (A) and (B). Connection identifiers include linksequence number 8500 and sequence element I.D. number 8510. Position (C)is used to specify electronic links to electronic reference information.

DP4 of FIG. 70 has a link sequence number 8500 of "1." Thus the imageinformation of document portion DP4 belongs to a first linked sequence.A sequence element I.D. number is not shown which indicates that thedefault sequence element I.D. number of "1" is assigned to DP4. Documentportion DP5 is also part of Linked Sequence "1". It has a sequenceelement I.D. number of "2," and thus is the second member in linkedsequence "1." Note that the sequence number of DPS is "1" and not "2." Asequence number of "2" would signify that DP5 is part of a second linkedsequence as opposed to the first linked sequence. DP6 is a secondexample of a document portion belonging to a first linked sequence, inwhich it is the second member of the sequence. DP6 also contains anelectronic link to electronic reference information. The electronicreference information associated with DP6 may be acquired in the mannerpreviously described.

The sample document image processing language described in connectionwith FIG. 70 also supports the case where the electronic referenceinformation to be linked to a portion of the document image is in theform of optically encoded data imprinted on hardcopy, where the hardcopyis received as part of the incoming document stream. Using DP3("Gates-Mail", or G-Mail) for illustration, the portion of the documentimage making the call to electronic reference information is marked asfollows: locus-C, 8520, is provided with the number of the piece ofelectronic reference information to be linked to that portion of thedocument image, e.g. "1." Each portion of optically encoded data ismarked as follows: locus-D, 8490, is marked with an "X." The symbol "X"has been selected to designate optically encoded information. Locus-A,8500, is marked with a numeral which matches the current piece ofelectronic reference information, e.g. "1". Therefore, locus-A for theoptically encoded data portions matches Locus-C for the portion of thedocument image to be electronically linked to the optically encodeddata; locus-B is marked with the number of the particular portion ofoptically encoded data--for example "1" if there is only one opticallyencoded portion, and "1" and "2" respectively in association with thefirst and second portions, where there are two optically encoded dataportions. This is illustrated and described further below in connectionwith FIG. 72.

The foregoing illustration represents one possible embodiment of adocument image processing language of the present invention. Otherembodiments are possible. It will also be appreciated by persons skilledin the art that some of the methods taught herein may be extrapolated todocument images specified using page description languages.

FIGS. 71(a) and 71(b) show that the delimiters graphically added todocument images may be electronically erased from the electronicrepresentations of the images they appear in. That is, the graphicalidentifiers added to hardcopy need not appear in their receivedversions. Shown in FIG. 71(a) are the pixels of delimiters 8830 and the"1" bits 8850 which code for them. (Single bracket delimiters aredepicted in the drawing for simplicity) After the delimiters have beenfound by pattern matching performed on the electronic representation ofthe document image, the "1" bits 8850 which make up the delimiter pixelscan be translated to "0" bits 8860, yielding clear, or background colorpixels 8840. Auto-erasure of delimiters also demonstrates thatdelimiters are not always necessary. In many instances document portionsmay be designated by a predetermined attribute of the portion. Forexample, the presence of a pixel density (shade) greater than apredetermined level may, by agreement between sender and recipient,denote an area to be translated into red. Thus, rather than usinggraphical delimiters to designate a portion of text and another symbolto identify a transfer function, it is possible to simply define pixeldensities as representing different colors within the document imageprocessing language. In this manner, no symbols need be added to thedocument--it is only necessary to alter the appearance of the textitself in a predetermined manner.

FIG. 72 provides two examples of how optically encoded data present inthe transmitted document stream may be specified using the sampledocument image processing language disclosed herein. The opticallyencoded data may comprise audio or video data in optically encodedformat. In FIG. 72, 8880 represents a first portion of a document imageto be electronically linked to electronic data appearing in opticallyencoded form. Because 8880 is the first portion to be so linked, 8940 ismarked with a "1". Because 8880 is to be electronically linked tooptically encoded data which needs to be translated into electronicformat in order to be of use, 8950 is marked with an "X". The actualoptically encoded data is comprised of 8890 and 8900. To designate 8890as a piece of optically encoded data, an "X" is placed next to the lowerright corner of the delimited portion, 8980 containing the opticallyencoded data. To designate 8890 as a piece of optically encoded dataassociated with the first portion (8880) of the document image having anelectronic link to optically encoded data, 8960 is marked "1".Similarly, 9000 is also marked with a "1" to designate 8900 as anotherpiece of optically encoded data associated with the portion 8880 of thedocument image. To identify 8890 as the first of two parts of a singleunified piece of optically encoded data, 8970 is marked with a "1". Thesample image processing language of the present invention permits eachpiece of optically encoded data to be comprised of a plurality of parts.In the example in FIG. 72, the optically encoded data is comprised oftwo parts: 8890 and 8900. The second part 8900 of the single piece ofoptically encoded data associated with document portion 8880 is markedwith a "2", 9010, in the lower left of the image portion containing8900. The second example of FIG. 72 can be seen to parallel the first,just-described, example. In the second example, document image portion8910 is to be electronically linked to a piece of optically encoded dataalso comprised of two pieces: 8920 and 8930. Note however, that because8910 represents the second portion of the document image to beelectronically linked to optically encoded data, 9020 is marked with a"2". For corresponding reasons, 9050 and 9080 are also marked with a"2".

In FIG. 73 a handwritten hardcopy document, 10010, is shown being fedinto the document scanner of an ordinary facsimile machine, 10012.Document 10010 contains portions to be electronically linked toelectronic reference information. In the present illustration theelectronic reference information is comprised of instructions and/ordata which may be used to establish a connection with a remote location.As discussed earlier, whereas a variety of predetermined attributes ofthe document image may serve to identify electronically linkableportions, in FIG. 73, delimiters are shown.

FIG. 74 illustrates one embodiment of a fall-back option under the DualDisconnect Protocol (D₂ P) described in connection with FIGS. 48 and 58.In FIG. 74 the sending station has already finished transmitting theelectronic graphical representation of the document image, and is nowattempting to negotiate an optimum mode for the automated transmissionof electronic reference information to the receiving station. Theembodiment shown represents but one possibility and is not meant to belimiting. After the sending station transmits the DCN signal (whichwould cause a conventional receiving fax apparatus to hang up) it sendsthe DTMF "*" tone of 941 Hz+1209 Hz, followed by CNG which consists ofan 100 Hz tone that is intermittently on for 0.5 seconds and off for 3seconds. The receiving apparatus responds by sending "VOA" to indicatethat it is capable of receiving electronic reference information in anautomated session from the sending station. A specific setup may then benegotiated between sending and receiving stations based on thecapabilities common to both sets of equipment at the two locations. Theelectronic reference information may then be transmitted. These conceptswould be appreciated by persons skilled in the art of handshakingprocedures between pairs of communicating data transmissions devices.Also, while the sending station is depicted as sending a DCN signalfollowed by the "*" tone, this is not mandatory. It is not a requirementfor the receiving station to operate in "Silent Answer Mode". Inalternate embodiments the receiving apparatus may issue a response suchas a tonal reply or voice message following receipt of the "DCN" signalfrom the sending station.

FIG. 75 illustrates an example of a voice menu generated by theapparatus at the receiving station to assist an operator at the sendingstation through the process of creating electronic reference informationfor a fax document. The voice menu system of FIG. 75 may be used even inthe case where a fax document and its associated electronic referenceinformation are transmitted using Simultaneous Voice-Data (SVD)technology. However, in FIG. 75 it is depicted within the context of theDual Disconnect Protocol (D₂ P), because that communications schemeoffers backwards compatibility with the prior art such as conventionalfacsimile machines operating on the public switched telephone network.It should be understood that the present invention is not constrainedwith respect to the temporal sequence in which a fax document and itsassociated electronic reference information is transmitted. In FIG. 75,after receiving the "*" tone, the receiving apparatus may issue a seriesof messages 10059 as part of its voice menu, beginning with theannouncement: "Entering Fax Phase II". This announcement, of course, isfurnished for the sake of example and should not be construed aslimiting. When the person at the sending station hears the desiredoption in the voice menu, that choice may be selected, 10063, bypressing the appropriate key on the telephone keypad. These practicesare well known in the art of voice menu driven systems.

FIG. 76 shows some of the voice menu options. For example, to createnotes--including voice notes, one would press the "1" key to enter thenotes menu, and then the "1" key again to access the audio sub-menu.Within the audio sub-menu additional instructions are furnished toassist a user through the process. 10087 shows part of the notes menu.10089 shows some of the menu choices offered for the "Talk-Back"feature. The "Talk-Back" feature allows the recipient of a fax documentto automatically establish a connection with a remote site, by simplyselecting (such as by clicking on . . . ) the portion of the faxdocument making the reference to the remote site. For example, if theremote site happens to be someone's voice telephone number, thattelephone number will be automatically dialed by the apparatus 1000. Ifthe remote site happens to be a WEB site on the Internet, the apparatus1000 will establish a connection to the Internet and then download theindicated information. 10089 in FIG. 76 also shows that, for example,the key sequence "2" and then "2" and then "2" again permits an operatorat a sending station to designate a number for a fax-on-demand server.

FIG. 76 illustrated how an operator at a sending station could use thevoice menu system provided by the apparatus at the receiving station tomanually select and then input electronic reference information to thereceiving apparatus. FIG. 77 shows how a telephone key pad 10140 may beused to pass alphanumeric information onto the receiving apparatus.Methods for communicating alphanumeric information using a standardtelephone key pad are known in the art. The e-Mail address:"sjr@nasa.org", 10162, may be keyed in by the sequence 10165. Similarly,the Internet Web address 10169 may be input using the sequence 10167.Also, while it is common practice to employ "dual-digit encoding" ininputting alphanumeric information using a telephone keypad, forinstance, the letter "s" is comprised of the key sequence 73 because "s"appears on the "7" key and is the 3rd choice out of the lettersappearing on the "7" key: "PRS"--other arrangements are also possible.For example, triple-digit encoding may be used. It is also possible forembodiments of the receiving apparatus capable fo voice recognition toaccept address information from the sending station in the form of aspoken human voice.

FIG. 78 depicts the handwritten paper document 10010 of FIG. 73 on thescreen of the receiving apparatus. In the on-screen version of thedocument 10205, are four regions that are electronically linked toelectronic reference information: 10220, 10223, 10225 and 10227. Theseregions correspond to the portions of the original that possessed thepredetermined attribute designating them as link regions. This can beseen in FIG. 73, where the names Mary, Steve, Sonia and Susan wereenclosed in delimiters. Whereas link regions 10220, 10223, 10225 and10227 may be displayed using highlight boxes, in FIG. 78 they areinstead displayed in color, for example, in green. The method foraccomplishing this, namely by remapping screen display attributes, hasbeen taught above (See FIGS. 63, 64, 66, 68, 69, 71). Therefore, a usermay designate just how linked regions are to be displayed on the screen;which is not always in highlight boxes. In FIG. 78 a cursor 10255, inthe form of an arrow, is pointing to the name "Mary", 10227, which islinked to electronic reference information comprised of contactinformation. Whenever the cursor passes over a linked region itsattributes are displayed. In the case of 10227, the portion of thedocument image containing the name "Mary" makes use of the "Talk-Back"feature of the present invention. Therefore, by double clicking on thename "Mary" a recipient of fax document 10010 can have the receivingapparatus automatically establish contact with, for instance, Mary. The"What box" 10215 identifies the electronic reference informationassociated with the region "Mary", 10227, as that of a voice telephonenumber: "Telephone.Voice". The receiving apparatus knows this becausethe sending station, in using the "Talk-Back" feature 10089 in FIG. 76,made a selection from the menu which identified the input data as being"For Voice Telephony". The "Where-Box" 10210 shows the contact address,for example the voice telephone number: 535-3665. By pressing the "Open"button in the menu 10206, an operator will cause the receiving apparatusto establish a connection to the "Talk-Back" address, which in thepresent example, means dialing the voice telephone number 535-3665, andthen automatically turning on the microphone and speaker of thereceiving apparatus (1000). A "Talk-Back" address can be other than avoice telephone number. For example, in FIG. 78 it may also be a faxnumber such as 10236, a beeper number such as 10233, or an e-Mailaddress such as 10235, etc..

FIG. 79 shows how a "Talk-Back" connection can be made from theConnection Processor window. The option of using the ConnectionProcessor window can be selected by a user from the "Preferences"option, which is accessed from the "Mode" button in the menu bar 10206.When the cursor 10255 passes over a linked region such as 10277, theattributes of the link region are displayed, as in 10210 and 10215. Ifthe cursor is then moved to another region of the document image that isfree of linked information, the boxes 10210 and 10215 will be blank.However, by clicking once over a linked region the address associatedwith that region will be loaded into the information boxes 10210 and10215, so that if the cursor then moves off of that linked region, andonto a link-free area, the already-loaded information will remaindisplayed in the boxes 10210 and 10215. A user may then click on the"Open" button in the menu bar 10206 to bring up window 10258 into theConnection Processor. Alternately (if this mode has been set), doubleclicking on a link region having a contact address will open theConnection Processor window 10258. It is also possible to select asetting whereby double clicking on a region having a contact addresswill directly cause a connection to be established, without theintervening Connection Processor window popping up. In FIG. 79, theConnection Processor window 10258 displays the contact address, or"Target" 10247, as well as its "Type" 10249.

FIG. 80 shows an example where a connection will be established with analphanumeric pager. In the handwritten letter, 10205 of FIG. 80, thereis one person: Sonia 10223, who is to be contacted through her pagingunit. When cursor 10255 is placed over the name "Sonia": 10223, theWhat-Box 10215 displays "Pager.Alphanumeric" to indicate that theaddress of "617.879.4321 ", which is displayed in the Where-Box: 10210,is for an alphanumeric paging unit. Pressing "Open" 10287 in the menubar 10206 will bring up the Connection Processor window 10258. IfSonia's paging unit had been of the straightforward numeric-onlyvariety, a connection would have been directly established--that is, theapparatus would have transmitted its own number to Sonia's device.However, because Sonia's unit is an alphanumeric pager, the ConnectionProcessor window opens to let the operator input text to be sent toSonia's alphanumeric pager. This is illustrated in FIG. 81.

Double clicking on "Sonia" 10223, or single clicking on "Sonia" 10223and then pressing the "Open" button in menu bar 10206 causes theConnection Processor window 10258 to pop up in FIG. 81. This time, the"Type" identifier 10249 displays "Alphanumeric Pager", and the "Target"identifier 10247 shows the pager's number. Text entry box 10264 isprovided for entering text to be sent to the paging unit. Note that thecursor 10255 transforms itself into an I-Beam 10269 when it is situatedwithin the text entry box 10264. Pressing the "Yes" button 10273 willcause the text entered in the text entry box 10264 to be sent to thealphanumeric pager at the "Talk Back" address specified.

FIG. 82 illustrates a handwritten letter 10510 that includes a"Talk-Back" address in the form of an Internet World Wide Web address.In FIG. 82 an individual has pasted a business card 10520 to ahandwritten letter 10510. That business card includes references to anordinary telephone number, a fax-on-demand number, and the URL of acompany's web site. As a convenience to the intended recipient of theletter, the author of the letter wishes to make use of the "Talk-Back"feature of the present invention to enable the receiving apparatus toautomatically contact the company named in the attached business cardfor the addressee. The linked portions of the fax letter are chosen tobe 10550 and 10530, because these are the portions of the attachedbusiness card which display contact information. They are designatedusing delimiters 10555, although, as mentioned repeatedly above, this isnot the only means for designating linked portions.

In FIG. 83 the letter 10510 is sent using a conventional facsimilemachine 10012; the pertinent "Talk-Back" information is then keyed inusing the telephone keypad 10140. The Web address "www.pie.com", 10165,is entered as the sequence of numbers depicted as 10167. The fax number10171 is entered as the sequence of numbers 10173. And, similarly, thevoice phone number 10175 is input as the numeric sequence shown as10177.

FIG. 84 shows how the business card 10520 of FIG. 82 appears on thescreen of the receiving apparatus. Notice that the delimiters 10555 havebeen auto-erased as taught in connection with FIG. 71, and thus do notshow up on 10521. When cursor 10255 moves over region 10551, the"Where-Box" 10210 reveals that the "Talk-Back" address is that of a Website: "www.pie.com". The "What-Box" 10215 indicates that the region10551 possesses two "Talk-Back" addresses, and that the web address of"www.pie.com" is only the first of the two: " 1 of 2!". The "What-Box"10215 further shows that the web address "www.pie.com" is the only webaddress present "(1/1)". For example, had there been another web addressreference, the "What Box" 10215 would have instead showed " 1 of 3! Web(1/2)". The business card 10521 in FIG. 84 also includes a reference toa voice telephone number, which is associated with region 10531. Noticethat an up arrow 10277 and a down arrow 10278 have appeared adjacent tothe "What" and "Where" boxes. These permit a user to scroll to specific"TalkBack" addresses when there are a plurality of such addressesassociated with a particular region of the document image. If the userwere to press the "Open" button 10287 now, a connection to "www.pie.com"would be established by the Connection Processor in fulfillment of the"Talk-Back" request made by the user. Thus a fax document, which mayeven be handwritten, and which may be transmitted using an ordinaryfacsimile machine, can allow the recipient of the document to establishcontact with a Web site by simply selecting the portion of the receivedfax document that refers to the Web reference.

FIG. 85 shows the use of the down arrow 10278 to scroll to the second ofthe two "TalkBack" addresses linked to region 10551. The cursor 10255 isillustrated on top of the down arrow 10278. The "What-Box" 10215accordingly indicates that the second "Talk-Back" address supplies aFax-On-Demand number. Pressing "Open" 10287 will cause a connection tobe established to the Fax-On-Demand server at the number indicated in"Where-Box" 10210.

FIG. 86 shows another example involving a "Talk-Back" address comprisedof a Web site. Consider the following scenario. A person sees anadvertise for a theatre in the bus on the way to work. Anxious to obtainshow information, she calls while still on the bus, using her cellularphone, the number displayed in the ad. The theatre's fax-on-demandserver answers her call and processes her request for information, in amanner well known in the art of fax-on-demand devices. She gives it thenumber of her fax tablet (i.e. device 1000 of FIG. 41) which is sittingin her office charging. By the time she arrives at work, document 12210shown in FIG. 86 has already been received by her fax tablet. Indocument 12210 is an icon, 12220, depicting a butler, or "server", seenholding a tray containing a computer--which is serving documents. Thisicon may be clicked on to retrieve from the theatre's web site asymbolically encoded version of document 12210; which may be desirablein instances where the received fax document is not completely clear.Thus clickable icon 12220 furnishes the option of obtaining an "Internetcopy" of the document, when they are available. Highlight boxes 12231and 12233 are regions which enable automatic connections to be made tovoice telephone numbers, via the "Talk-Back" feature of the presentinvention. 12237 re-establishes contact with the fax server whichoriginally sent document 12210. And, 12235 generates a connection to an"STS" server (Station-To-Station Server), which is discussed in aco-pending patent application. The box labeled "December", 12215 in FIG.86, contains a link to the theatre's web site which is represented as12240. Therefore, by clicking on 12215 a user may access the web page12240. Within the web page 12240 are buttons 12251 for seeing videosequences of sneak show previews; 12253, for obtaining ticketinformation; and 12255, which retrieves a seating plan for the theatre.The just-mentioned concepts which relate to operation solely within thecontext of the Web are known in the art, and, in the absence of regardfor connectivity from an ordinary fax document, are also extraneous tothe object of the present invention.

Because the apparatus 1000 may be operated as a portable unitcommunicating over a cellular connection (see FIG. 43a), it is desirablethat the cellular connection be as resistant to cloning as possible.FIGS. 87 and 88 illustrate one embodiment of a system using "DynamicPassCode Protection" to hinder cellular cloning. In particular, FIGS. 87and 88 depict a specific embodiment of Dynamic PassCode Protection usinga "Dual Component" approach.

In FIG. 87, a portable apparatus 1000 is establishing a connection withthe nearest transmission station 13107 of a cellular service provider.In current art systems identification information is transmitted by themobile unit to identify itself to the cellular service provider. Morerecently, in addition to the identification information, a PIN number isalso transmitted. With the Dual Component Dynamic PassCode approachtaught herein, the conventional PIN number is replaced by a PassCodewhich is the equivalent of the PIN number, except that the PassCode ischanged immediately following the successful establishment of a cellularconnection. Therefore, immediately following establishment of aconnection, the just-used PassCode is invalidated. Thus if it isintercepted by a cellular thief it will no longer be of use. Thealgorithm applied to generate the dynamic passcode is contained in thefirmware of the apparatus 1000; it is also known by the computer of thecellular service provider so that both may generate new passcodes insynchrony. A multiplicity of algorithms may be available such that anytwo portable units 1000 have a low statistical likelihood of havingfirmware containing the same algorithm. Also, the algorithm is containedin the firmware of the apparatus 1000 so that it will be more difficultto re-program with a new algorithm. Only the manufacturer of theapparatus 1000 knows which algorithm is contained in any given unit,although the necessary data associating each unit with its algorithm ispassed on to the cellular service provider. It is further possible foreach apparatus to include multiple algorithms, one of which may bedesignated as active at the time cellular service is first initiated. Asshown in FIG. 87, the PassCode is generated preferably using twofunctions: "f" and "g". First, a dynamically changing variable isgenerated based on a function "g" of a number referred to as the "Co-PINNumber". The actual PassCode itself is then generated based on a secondfunction "f" of the user's PIN Number and the dynamic variable g(Co-PIN#). Thus PassCod=f(PIN #, g Co-PIN #!). The dynamic variable may befurnished by a number generator (i.e. function "g") which uses theCo-PIN# as its "seed", for instance. Variations on the theme are alsopossible, for example, using g(PIN#, Co-PIN#) either with anotherfunction "f" or alone, etc. It is also possible for the PassCode to becalculated using only the Co-PIN#.

FIG. 88 shows one method of synchronizing the Co-PIN Number between acellular service provider and a cellular customer. It is preferred thata cellular customer call, using a conventional wired line, a telephonenumber reserved for establishing Co-PIN Numbers by the cellular serviceprovider (e.g. 1.800.GetCode). Upon reaching a computer at the callednumber, the customer would provide information such as the phone numberof the portable unit and the PIN #, and obtain in return from thecomputer, a new Co-PIN #. This may be done as frequently as desired.However, although it is not preferred, an optional method for obtainingCo-PIN numbers using a cellular connection is illustrated in FIG. 88.

In FIG. 88 a portable apparatus 1000 receives a "Secure Envelope" 13126,from the nearest transmitting station 13107 of the cellular serviceprovider. The Secure Envelope contains the new Co-PIN Number 13133. Inorder to "open" the Secure Envelope 13126, the customer's PIN Number13129 is required. The Secure Envelope may comprise any suitable schemeknown in the art of data encryption for encrypting, or for otherwisecontaining or conveying the customer's Co-PIN # 13133, such that the PINNumber may be applied to extract the Co-PIN #.

The method shown in FIGS. 87 and 88 therefore offers a dynamic PassCodein place of the static PIN number currently used in the art.Additionally, the algorithm used to generate the dynamic passcode varieswith the apparatus so that even knowledge of the correct PIN# andCo-PIN# is insufficient. Further, the customer's PIN# is nevertransmitted over the airwaves, and so is kept confidential. Stillfurther, even when the Co-PIN# is transmitted over a cellularconnection, it is contained within a Secure Envelope so that itsinterception alone is inadequate.

The presently disclosed embodiments are to be considered in all respectsas illustrative and not restrictive, the scope of the invention beingindicated by the appended claims, rather than the foregoing description,and all changes which come within the meaning and range of equivalencyof the claims are therefore intended to be embraced therein.

What is claimed is:
 1. An apparatus for retrieving Internet informationusing facsimile data received with a facsimile reception protocol,comprising:means for receiving facsimile data using a facsimilereception protocol, said facsimile data containing a portion linked toan Internet address; means for obtaining Internet address informationfrom said portion of received facsimile data; means for automaticallyapplying said obtained Internet address information for establishingcommunication with said Internet address; means for retrievinginformation from said Internet address; and means for displaying atleast one of the received facsimile data and the retrieved Internetinformation.